How MongoDB's Journaling Works

MongoDB journal provides durability in the event of a failure, MongoDB uses write-ahead logging to on-disk journal files.

It has explained in nicely in below article.  Credit should go to the original author 

https://www.mongodb.com/blog/post/how-mongodbs-journaling-works

Cassandra data retrieving issue - Due to huge tombstones

Recently, one of the development team has complained about their Cassandra cluster isn't returning the data from one of the tables and soon after that issue faced cluster went down.

This incident happened over the weekend and on-call Cassandra DBA was informed and asked to check the cluster. Initially, Dev team has complained about the performance issue with Cassandra cluster, in other words, Cassandra cluster couldn't handle the current load( Later they have identified it was a sudden huge data load came directly from one of the subsystems)

As a first step, DBA decided to start the cluster by starting Cassandra service but he observed the Cassandra log saying "unable to handle the load" exception ( This not the actual error)

As usual heap size wasn't enough for the existing servers due to that new nodes have been added with 12 GB of heap size with total 16 GB of physical RAM.

Even after adding new nodes, DBA has executed repair command and the problem got solved temporarily. 

Late part of the day, Again Dev team has reported their issue still exist and ask DBA support. This time we have observed my WARN message for tombstoned and it was a huge error list. <-- This error was happening from the beginning but it didn't notice because the team wanted to start the cluster running state somehow.

With the high number of tombstones, we suspected that their might have some unusual behavior in application level and we checked with Dev team regarding application involvement with showed table.

Finally, the dev team has confirmed that they are using this table as a temporary location for particular data and delete the data in every 15 mins via corn job.

Since default grace period of the tombstone is nearly 10 days, data is not being deleted from the Cassandra cluster completely until grace period is over.  Due recursive call of the delete operation, these tombstones were slowly growing up and no repair jobs were running to resolve the data scatter.

With all of this,  select query on this table was unable to receive the data as it has to read the huge list of tombstones value to find a live or actual data and heap memory wasn't enough to load all tombstones values. 

As a solution, we have changed the "gc_grace_seconds" to the 1200( 20 mins) and upgrade the sstable and then compact the table manually in each node

connect to the instance and change the gc_grace_seconds value in gradebeforeitem  table

use xxxxxx_prd;

alter table gradebeforeitem with gc_grace_seconds = 1200;

Each command has executed in each node individually ( One after another) to reduce the impact.

nohup nodetool upgradesstables -a xxxxxx_prd   gradebeforeitem &

nohup nodetool compact xxxxxx_prd gradebeforeitem &

Once we received the confirmation and success story from dev team we reverted back to the " gc_grace_seconds " value to the default (864000 ) and set up the repair job in each node to run weekly. We scheduled the job keeping 24 hours difference between jobs to avoid the clashes between the repair jobs

Upcoming Features in MongoDB 4.0

As per the MongoDB documentation,  MongoDB will be introduced "Multi-Document Transactions".  MongoDB 4.0 will add multi-document transactions for replica sets.

You can find example here :
https://docs.mongodb.com/manual/upcoming/

MongoDB Cloud Manager vs. Ops Manager

• Enterprise Advanced comes with both Ops Manager and Cloud Manager so it is up to you which one you would like to use 

• Cloud Manager is managed software-as-a-service where you only have to install relevant agents (monitoring, automation, or backup) in your infrastructure. Ops Manager is on-premise software that is part of an Enterprise subscription: you have to install and manage all services on your own infrastructure.

• They are similar in terms of main feature areas (monitoring, automation, backup) and user interface but not identical. Since Cloud Manager is a managed service it is updated more regularly than Ops Manager releases and currently has some additional features such as integration with cloud provisioning instances via providers like AWS or Azure.

• Please note that the agent versions also differ between Cloud and Ops Manager, so you should ensure you are running the correct agents and referring to the relevant documentation.

MongoDB Enterprise Licence Pricing

Recently, I got a chance to have a call with one of the sales executive and found out below pricing information for MongoDB enterprise node

Please note that this cost only for MongoDB Enterprise binaries and once you purchased binaries you will be getting many features along with binaries.

There are costing only for data nodes. As an example, you don't need to pay for config servers or MongoS severs in MongoDB shard cluster.

As per the April 2108

• Production node is $12,990 per annum
• UAT/QA node is $ 6,495 per annum ( 50% from production cost)
• Development node is fee

MongoDB Enterprise version comes with trial feature and it has only 30 days. There are no features cut down or being stopped after the trial period.  But you are not allowed to use Enterprise more than 30 days without purchasing for the Production environment

Cost for 5 nodes production replica set runs for year,

5 x $12,990 = $ 64, 950

Since MongoDB didn't share this pricing information on their website, they could have a change pricing time to time. But you could estimate at least before reaching them.

About MongoDB lock file (mongodb.lock)

the mongodb.lock file is a simple file which holds a random number and it is doing
most important tasks for MongoDB

Important of mongodb.lock file

1. To detect unexpected/ Unclean shutdown

2. dbpath only access by ONE mongod process


If you have seen any unexpected shutdown or hanged MongoDB server you have to repair your MongoDB before it starts.

Here are basic steps

1. look at the mongodb log file and see bottom lines of log. You will see some clue on mongod hanged
    tail -n 20 /var/log/mongodb/mongodb.log
    you  may see below error message
    "Detected unclean shutdown - mongod.lock is not empty."

2. Create a backup copy of the data files in the --dbpath.
    mongodump --dbpath /data/mongodb/ -o /var

3. Start mongodb with --repair command
    mongod --dbpath /data/db --repair

After the repair, the dbpath should contain the repaired data files and an empty mongod.lock file. 

How life works, referring to Cells, Cell division, DNA, Genes, Chromosomes, and proteins.

 Nothing related to the database but it something worth knowing about our life: 

This article all about explores how life works in terms of basics biological factors. I’m going to discuss here Cell, Cell division, DNA, Genes, Chromosomes, and proteins. We start from the smallest structural and functional unit of an organism called the cell. As an example, the human body contains trillion of the cell and in another word, it’s a structural, functional and biological unit of all organisms. The cell provides structure for the body, take in nutrients from food, convert those nutrients into energy, and carry out specialized functions. Physically cells always have a boundary membrane and a membrane-bound structure containing biomolecules, such as nucleic acids, proteins, and polysaccharides. According to the structure of the cell, biologists divide organisms into two group. Those groups are,

●        Prokaryotes: prokaryotes cells surrounded by a membrane and a cell wall, with a circular strand of DNA contains their genes and it does not have a nucleus

●        Eukaryotes: eukaryotic cells contained within the nuclear envelop and separated from the cytoplasm. Nevertheless, these cells boast their own personal “power plants”, called mitochondria. These tiny organelles in the cell not only produce chemical energy but also hold the key to understanding the evolution of the eukaryotic cell.

Cell division is the process of a parent cell divides into two or more daughter cells and it occurs as part of the large cell cycle.

There are two distinct types of cell division in eukaryotes,

●        Vegetative division: daughter cells is generally identical to the parent cell(mitosis)

●        Reductive cell division: number of chromosomes in the daughter cells is reduced by half, to produce haploid gametes(meiosis)

In terms of definition of the Meiosis is “A type of cellular reproduction in which the number of chromosomes is reduced by half through the separation of homologous chromosomes, producing two haploid cells.” and Mitosis is “A process of asexual reproduction in which the cell divides in two producing a replica, with an equal number of chromosomes in each resulting diploid cell.” Prokaryotes are much simpler in their organization than eukaryotes. There are great many more organelles in eukaryotes, also more chromosomes. The usual method of prokaryote cell division is termed binary fission. The prokaryotic chromosome is a single DNA molecule that first replicates, then attaches each copy to a different part of the cell membrane. When the cell begins to pull apart, the replicate and original chromosomes are separated. Following cell splitting (cytokinesis), there are then two cells of identical genetic composition (except for the rare chance of a spontaneous mutation).

DNA (Deoxyribonucleic acid)

DNA contains the biological instructions that make each species unique. DNA, along with the instructions it contains, is passed from adult organisms to their offspring during reproduction. In another word, it looks like a blueprint for building different parts of the cell. Most of DNA is found inside a special area of the cell called the nucleus. Apart from the DNA located in the nucleus, humans and other complex organisms also have a small amount of DNA in a different structure called mitochondria. A nucleotide is the chemical building blocks of the DNA and it contains three parts, a phosphate group, a sugar group and one of four types of nitrogen bases. The four types of nitrogen bases found in Nucleotides are adenine (A), thymine (T), guanine (G) and cytosine (C). The order or sequence, of these bases, determine what biological instructions are contained in a strand of DNA. For example, the sequence ATCGTT might instruct for blue eyes, while ATCGCT might instruct for brown. Nucleotides are arranged in two long strands that form a spiral called a double helix. The structure of the double helix is more likely a ladder. The important property of DNA is that it can replicate itself. Each strand of DNA in the double helix can serve as a pattern for duplicating the sequence of bases. This is critical when cells divide because each new cell needs to have an exact copy of the DNA present in the old cell.

Genes

In term of biology, “A gene is the basic physical and functional unit of heredity. Genes, which are made up of DNA, act as instructions to make molecules called proteins.” In humans, genes vary in size from a few hundred DNA bases to more than 2 million bases but when it comes to general, genes carry the information that determines traits, which are features that are passed from parents. Genes are found on tiny spaghetti­like structures called chromosomes. The DNA also contains large sequences that do not code for any protein and their function is not known. The gene of the coding region encodes instructions that allow a cell to produce a specific protein or enzyme. There are nearly 50,000 and 100,000 genes with each being made up of hundreds of or thousands of chemical bases. Chromosomes

Chromosomes are the place where DNA is located since the cell is very small and organisms have many DNA molecules per cell, each DNA molecules must be tightly packaged and this package is a form of the DNA is called chromosome. Chromosomes come in matching sets of two (or pairs) and there are hundreds or thousands of genes in just one chromosome. Chromosomes in humans can be divided into two types those are, autosomes and sex chromosomes. Certain genetic traits are linked to a person's sex and are passed on through the sex chromosomes. The autosomes contain the rest of the genetic hereditary information. All act in the same way during cell division. Human cells have 23 pairs of chromosomes (22 pairs of autosomes and one pair of sex chromosomes), giving a total of 46 per cell. Half of these chromosomes come from one parent and half come from the other parent In addition to these, human cells have many hundreds of copies of the mitochondrial genome. Sequencing of the human genome has provided a great deal of information about each of the chromosomes.

Proteins

Proteins are a complex and large molecule that play critical roles in the body and proteins are made with hundreds or thousands of smaller units called amino acids, which are attached to one another in long chains. In order to make proteins, the gene from the DNA is copied by each of the chemical bases into messenger RNA (ribonucleic acid) or mRNA. The mRNA moves out of the nucleus and uses cell organelles in the cytoplasm called ribosomes to form the polypeptide or amino acid that finally folds and configures to form the protein.

All of the above factors are playing such a complex roles/ responsibilities to perform life with many more other things.