What is MongoDB FTDC  (Full Time Diagnostic Data Capture)? How does it impact your platform stability?

In simple words, MongoDB Inc. captures server behavior and stores it under a separate directory (data directory/diagnostic.data). This is mainly for MongoDB engineers, to process/analyze mongod and mongos when needed ( troubleshooting). FTDC data files are compressed and not human-readable. MongoDB Inc. engineers cannot access FTDC data without explicit permission and assistance from system owners or operators.

FTDC captures data every 1 second.

Sometimes this process leads to high resource consumption. In a few places, this process marked as overhead for MongoDB overhead.

i.e

https://bugzilla.redhat.com/show_bug.cgi?id=1834377

How ticket drops impacted this process,

This process/tread collects,

• serverStatus
• replSetGetStatus (mongod only)
• collStats for the local.oplog.rs collection (mongod only)

Once our servers are getting load, capturing every 1 second all the above data, could lead to holding tickets longer time than usual.  

The only drawback of disabling this feature is,  troubleshooting time may increase from MongoDB Inc. 

by default this feature is enabled.

How to disable it:

• Run time  ( Apply immediately but until next instance restart)
• config level ( You need to restart the instance)

Runtime 
use admin
db.adminCommand( { setParameter: 1 , diagnosticDataCollectionEnabled : false } )


MongoDB config file

setParameter:

  diagnosticDataCollectionEnabled: false

There is a huge change in our platform 

Disk reclaim on MongoDB WiredTiger Engine

This example shows that deleting MongoDB collection will release the disk space to OS ( only applicable for WT storage engine)

```
hasitha@DESKTOP / $ sudo du -sh /data/
6.0G    /data/

hasitha@DESKTOP / $ mongo_local
MongoDB shell version v3.4.21
connecting to: mongodb://localhost:27017/
MongoDB server version: 3.4.21
> show dbs
admin           0.000GB
central_db      0.000GB
e_1v6y03dpfs6z  0.001GB
e_2nas8av7dawh  5.552GB
e_da618q7mfzsv  0.001GB
e_p4rxh05m2k4w  0.008GB
e_testdb        0.001GB
e_yrrsst6ncktg  0.004GB
local           0.000GB
m_testdb        0.000GB
m_yrrsst6ncktg  0.000GB
s_testdb        0.000GB

> use e_2nas8av7dawh
switched to db e_2nas8av7dawh

> show collections
users

> db.users.drop()

true

> show collections

> exit

bye

hasitha@DESKTOP / $ sudo du -sh /data/

358M    /data/

```

Wildcard Indexing - The new feature comes in MongoDB 4.2

It's a new topic on the MongoDB world.  Here is the first blog from MongoDB Developer:
https://www.mongodb.com/blog/post/coming-in-mongodb-42--1-wildcard-indexes

Download MongoDB backup files (Encrypted) from S3

After spending a few hours, I have written a small script where you can download MongoDB backup files from S3. The specialty of this script is, this can download backup files which are encrypted by using key value ( Encrypted key)




 # DO NOT modify the encryption key unless if they used different key for encryption
 key="<key value>"
 # s3 path for MongoDB backups. You can find it from MongoDB backup logs ( You may not be able to find full s3 path from backup config file as backup file creates additional backup folder.
 s3path="< S3 path >"
 # Backup download location
 backupLocation="/data/backups/tmp"

 echo " ---------------------------------------------------------"
 echo " List of backup files store in $s3path "
 echo ""
 echo " ---------------------------------------------------------"
 echo -e "   Date  |   Time  |   Size  | \tName of the file"
 echo " ---------------------------------------------------------"
 aws s3 ls $s3path
 echo " ---------------------------------------------------------"
 echo " What is the database file name that you need to download ? < Please type or copy paste full name of the backup file here > "
 read filename
 if [ -z $filename ]
  then
   echo " ERROR:: No file name has entered !!! ABORTED"
   exit 1
 fi

 echo ""
 echo " $s3path$filename will be downloading to $backupLocation "
 aws s3 cp $s3path$filename $backupLocation
 echo ""
 echo " ---------------------------------------------------------"
 echo " Decrypting the file ..... "
 openssl enc -d -aes-256-cbc -in $backupLocation/$filename -out $backupLocation/${filename::-4} -k $key
 echo " ---------------------------------------------------------"
 ls -al $backupLocation/
 echo ""
 echo " ---------------------------------------------------------"
 echo ""
 echo " Where do you want to extract the backup files ? Default location is /data/backups/tmp"
 read restoreLocation
 if [ -z $restoreLocation ] 
  then 
   restoreLocation="/data/backups/tmp" 
 fi

 echo ""
 echo " Extracting db files to $restoreLocation "
 tar -xvf $backupLocation/${filename::-4} -C $restoreLocation

 echo " ---------------------------------------------------------"
 echo " list of files and directories in $restoreLocation "
 ls -al $restoreLocation/
 echo " ---------------------------------------------------------"
 echo " Remove downloaded file : $backupLocation/$filename"
 echo " ---------------------------------------------------------"
 rm -f $backupLocation/$filename
  echo ""
 echo " #########################################################"
 echo " # You have successfully downloaded backup files. Please #"
 echo " #  restore your files using mongorestore utility        #"
 echo " #########################################################"       

Delete your Large MongoDB database effectively

You can use below javascript code to delete large database without having a long duration of global lock( which is blocked all other operations on this database, maybe your instance might get hang).

Initially, this script will delete all indexes belongs to the selected collection and then its' documents one by one. Once the collection has no documents, it drops the collection finally. Between these operations, sleep function is configured to avoid the global lock being held in longer duration.

var dbname= "dummy"
var URL = "localhost:27017/"+ dbname
db = connect(URL);

var collections = db.getCollectionNames();

for (var i in collections) {

        print ("Deleting collection : " + collections[i] +"...");
        print ("Droping all indexes in "+ collections[i] + "...");
        db[collections[i]].dropIndexes();
        sleep(5000);
        db[collections[i]].find().forEach(function(mydoc) {
                print ("Deleting document : " + mydoc._id)
                db[collections[i]].remove({'_id': mydoc._id})
                //print(mydoc.name);
                sleep(1000);
                }
        )
        db[collections[i]].drop();
        sleep(5000);
}

function sleep(milliseconds) {
  var start = new Date().getTime();
  while (new Date().getTime()< start + milliseconds)  {
    // Do nothing
  }
}

MongoDB Memory Management

Nice Article: http://www.briancarpio.com/2012/05/03/mongodb-memory-management/

Credit goes to the Brian Carpio

How MongoDB's Journaling Works

MongoDB journal provides durability in the event of a failure, MongoDB uses write-ahead logging to on-disk journal files.

It has explained in nicely in below article.  Credit should go to the original author 

https://www.mongodb.com/blog/post/how-mongodbs-journaling-works