Disk reclaim on MongoDB WiredTiger Engine

This example shows that deleting MongoDB collection will release the disk space to OS ( only applicable for WT storage engine)

```
hasitha@DESKTOP / $ sudo du -sh /data/
6.0G    /data/

hasitha@DESKTOP / $ mongo_local
MongoDB shell version v3.4.21
connecting to: mongodb://localhost:27017/
MongoDB server version: 3.4.21
> show dbs
admin           0.000GB
central_db      0.000GB
e_1v6y03dpfs6z  0.001GB
e_2nas8av7dawh  5.552GB
e_da618q7mfzsv  0.001GB
e_p4rxh05m2k4w  0.008GB
e_testdb        0.001GB
e_yrrsst6ncktg  0.004GB
local           0.000GB
m_testdb        0.000GB
m_yrrsst6ncktg  0.000GB
s_testdb        0.000GB

> use e_2nas8av7dawh
switched to db e_2nas8av7dawh

> show collections
users

> db.users.drop()

true

> show collections

> exit

bye

hasitha@DESKTOP / $ sudo du -sh /data/

358M    /data/

```

Wildcard Indexing - The new feature comes in MongoDB 4.2

It's a new topic on the MongoDB world.  Here is the first blog from MongoDB Developer:
https://www.mongodb.com/blog/post/coming-in-mongodb-42--1-wildcard-indexes

Download MongoDB backup files (Encrypted) from S3

After spending a few hours, I have written a small script where you can download MongoDB backup files from S3. The specialty of this script is, this can download backup files which are encrypted by using key value ( Encrypted key)




 # DO NOT modify the encryption key unless if they used different key for encryption
 key="<key value>"
 # s3 path for MongoDB backups. You can find it from MongoDB backup logs ( You may not be able to find full s3 path from backup config file as backup file creates additional backup folder.
 s3path="< S3 path >"
 # Backup download location
 backupLocation="/data/backups/tmp"

 echo " ---------------------------------------------------------"
 echo " List of backup files store in $s3path "
 echo ""
 echo " ---------------------------------------------------------"
 echo -e "   Date  |   Time  |   Size  | \tName of the file"
 echo " ---------------------------------------------------------"
 aws s3 ls $s3path
 echo " ---------------------------------------------------------"
 echo " What is the database file name that you need to download ? < Please type or copy paste full name of the backup file here > "
 read filename
 if [ -z $filename ]
  then
   echo " ERROR:: No file name has entered !!! ABORTED"
   exit 1
 fi

 echo ""
 echo " $s3path$filename will be downloading to $backupLocation "
 aws s3 cp $s3path$filename $backupLocation
 echo ""
 echo " ---------------------------------------------------------"
 echo " Decrypting the file ..... "
 openssl enc -d -aes-256-cbc -in $backupLocation/$filename -out $backupLocation/${filename::-4} -k $key
 echo " ---------------------------------------------------------"
 ls -al $backupLocation/
 echo ""
 echo " ---------------------------------------------------------"
 echo ""
 echo " Where do you want to extract the backup files ? Default location is /data/backups/tmp"
 read restoreLocation
 if [ -z $restoreLocation ] 
  then 
   restoreLocation="/data/backups/tmp" 
 fi

 echo ""
 echo " Extracting db files to $restoreLocation "
 tar -xvf $backupLocation/${filename::-4} -C $restoreLocation

 echo " ---------------------------------------------------------"
 echo " list of files and directories in $restoreLocation "
 ls -al $restoreLocation/
 echo " ---------------------------------------------------------"
 echo " Remove downloaded file : $backupLocation/$filename"
 echo " ---------------------------------------------------------"
 rm -f $backupLocation/$filename
  echo ""
 echo " #########################################################"
 echo " # You have successfully downloaded backup files. Please #"
 echo " #  restore your files using mongorestore utility        #"
 echo " #########################################################"       

Delete your Large MongoDB database effectively

You can use below javascript code to delete large database without having a long duration of global lock( which is blocked all other operations on this database, maybe your instance might get hang).

Initially, this script will delete all indexes belongs to the selected collection and then its' documents one by one. Once the collection has no documents, it drops the collection finally. Between these operations, sleep function is configured to avoid the global lock being held in longer duration.

var dbname= "dummy"
var URL = "localhost:27017/"+ dbname
db = connect(URL);

var collections = db.getCollectionNames();

for (var i in collections) {

        print ("Deleting collection : " + collections[i] +"...");
        print ("Droping all indexes in "+ collections[i] + "...");
        db[collections[i]].dropIndexes();
        sleep(5000);
        db[collections[i]].find().forEach(function(mydoc) {
                print ("Deleting document : " + mydoc._id)
                db[collections[i]].remove({'_id': mydoc._id})
                //print(mydoc.name);
                sleep(1000);
                }
        )
        db[collections[i]].drop();
        sleep(5000);
}

function sleep(milliseconds) {
  var start = new Date().getTime();
  while (new Date().getTime()< start + milliseconds)  {
    // Do nothing
  }
}

MongoDB Memory Management

Nice Article: http://www.briancarpio.com/2012/05/03/mongodb-memory-management/

Credit goes to the Brian Carpio

How MongoDB's Journaling Works

MongoDB journal provides durability in the event of a failure, MongoDB uses write-ahead logging to on-disk journal files.

It has explained in nicely in below article.  Credit should go to the original author 

https://www.mongodb.com/blog/post/how-mongodbs-journaling-works

Cassandra data retrieving issue - Due to huge tombstones

Recently, one of the development team has complained about their Cassandra cluster isn't returning the data from one of the tables and soon after that issue faced cluster went down.

This incident happened over the weekend and on-call Cassandra DBA was informed and asked to check the cluster. Initially, Dev team has complained about the performance issue with Cassandra cluster, in other words, Cassandra cluster couldn't handle the current load( Later they have identified it was a sudden huge data load came directly from one of the subsystems)

As a first step, DBA decided to start the cluster by starting Cassandra service but he observed the Cassandra log saying "unable to handle the load" exception ( This not the actual error)

As usual heap size wasn't enough for the existing servers due to that new nodes have been added with 12 GB of heap size with total 16 GB of physical RAM.

Even after adding new nodes, DBA has executed repair command and the problem got solved temporarily. 

Late part of the day, Again Dev team has reported their issue still exist and ask DBA support. This time we have observed my WARN message for tombstoned and it was a huge error list. <-- This error was happening from the beginning but it didn't notice because the team wanted to start the cluster running state somehow.

With the high number of tombstones, we suspected that their might have some unusual behavior in application level and we checked with Dev team regarding application involvement with showed table.

Finally, the dev team has confirmed that they are using this table as a temporary location for particular data and delete the data in every 15 mins via corn job.

Since default grace period of the tombstone is nearly 10 days, data is not being deleted from the Cassandra cluster completely until grace period is over.  Due recursive call of the delete operation, these tombstones were slowly growing up and no repair jobs were running to resolve the data scatter.

With all of this,  select query on this table was unable to receive the data as it has to read the huge list of tombstones value to find a live or actual data and heap memory wasn't enough to load all tombstones values. 

As a solution, we have changed the "gc_grace_seconds" to the 1200( 20 mins) and upgrade the sstable and then compact the table manually in each node

connect to the instance and change the gc_grace_seconds value in gradebeforeitem  table

use xxxxxx_prd;

alter table gradebeforeitem with gc_grace_seconds = 1200;

Each command has executed in each node individually ( One after another) to reduce the impact.

nohup nodetool upgradesstables -a xxxxxx_prd   gradebeforeitem &

nohup nodetool compact xxxxxx_prd gradebeforeitem &

Once we received the confirmation and success story from dev team we reverted back to the " gc_grace_seconds " value to the default (864000 ) and set up the repair job in each node to run weekly. We scheduled the job keeping 24 hours difference between jobs to avoid the clashes between the repair jobs