Monday, September 2, 2019

Disk reclaim on MongoDB WiredTiger Engine

This example shows that deleting MongoDB collection will release the disk space to OS ( only applicable for WT storage engine)



```
hasitha@DESKTOP / $ sudo du -sh /data/
6.0G    /data/

hasitha@DESKTOP / $ mongo_local
MongoDB shell version v3.4.21
connecting to: mongodb://localhost:27017/
MongoDB server version: 3.4.21
> show dbs
admin           0.000GB
central_db      0.000GB
e_1v6y03dpfs6z  0.001GB
e_2nas8av7dawh  5.552GB
e_da618q7mfzsv  0.001GB
e_p4rxh05m2k4w  0.008GB
e_testdb        0.001GB
e_yrrsst6ncktg  0.004GB
local           0.000GB
m_testdb        0.000GB
m_yrrsst6ncktg  0.000GB
s_testdb        0.000GB

> use e_2nas8av7dawh
switched to db e_2nas8av7dawh

> show collections
users

> db.users.drop()
true

> show collections

> exit
bye
hasitha@DESKTOP / $ sudo du -sh /data/
358M    /data/

```


Monday, June 17, 2019

Wednesday, February 6, 2019

Download MongoDB backup files (Encrypted) from S3


After spending a few hours, I have written a small script where you can download MongoDB backup files from S3. The specialty of this script is, this can download backup files which are encrypted by using key value ( Encrypted key)




 # DO NOT modify the encryption key unless if they used different key for encryption
 key="<key value>"
 # s3 path for MongoDB backups. You can find it from MongoDB backup logs ( You may not be able to find full s3 path from backup config file as backup file creates additional backup folder.
 s3path="< S3 path >"
 # Backup download location
 backupLocation="/data/backups/tmp"

 echo " ---------------------------------------------------------"
 echo " List of backup files store in $s3path "
 echo ""
 echo " ---------------------------------------------------------"
 echo -e "   Date  |   Time  |   Size  | \tName of the file"
 echo " ---------------------------------------------------------"
 aws s3 ls $s3path
 echo " ---------------------------------------------------------"
 echo " What is the database file name that you need to download ? < Please type or copy paste full name of the backup file here > "
 read filename
 if [ -z $filename ]
  then
   echo " ERROR:: No file name has entered !!! ABORTED"
   exit 1
 fi

 echo ""
 echo " $s3path$filename will be downloading to $backupLocation "
 aws s3 cp $s3path$filename $backupLocation
 echo ""
 echo " ---------------------------------------------------------"
 echo " Decrypting the file ..... "
 openssl enc -d -aes-256-cbc -in $backupLocation/$filename -out $backupLocation/${filename::-4} -k $key
 echo " ---------------------------------------------------------"
 ls -al $backupLocation/
 echo ""
 echo " ---------------------------------------------------------"
 echo ""
 echo " Where do you want to extract the backup files ? Default location is /data/backups/tmp"
 read restoreLocation
 if [ -z $restoreLocation ] 
  then 
   restoreLocation="/data/backups/tmp" 
 fi

 echo ""
 echo " Extracting db files to $restoreLocation "
 tar -xvf $backupLocation/${filename::-4} -C $restoreLocation

 echo " ---------------------------------------------------------"
 echo " list of files and directories in $restoreLocation "
 ls -al $restoreLocation/
 echo " ---------------------------------------------------------"
 echo " Remove downloaded file : $backupLocation/$filename"
 echo " ---------------------------------------------------------"
 rm -f $backupLocation/$filename
  echo ""
 echo " #########################################################"
 echo " # You have successfully downloaded backup files. Please #"
 echo " #  restore your files using mongorestore utility        #"
 echo " #########################################################"       

Tuesday, February 5, 2019

Delete your Large MongoDB database effectively



You can use below javascript code to delete large database without having a long duration of global lock( which is blocked all other operations on this database, maybe your instance might get hang).

Initially, this script will delete all indexes belongs to the selected collection and then its' documents one by one. Once the collection has no documents, it drops the collection finally. Between these operations, sleep function is configured to avoid the global lock being held in longer duration.

var dbname= "dummy"
var URL = "localhost:27017/"+ dbname
db = connect(URL);

var collections = db.getCollectionNames();

for (var i in collections) {

        print ("Deleting collection : " + collections[i] +"...");
        print ("Droping all indexes in "+ collections[i] + "...");
        db[collections[i]].dropIndexes();
        sleep(5000);
        db[collections[i]].find().forEach(function(mydoc) {
                print ("Deleting document : " + mydoc._id)
                db[collections[i]].remove({'_id': mydoc._id})
                //print(mydoc.name);
                sleep(1000);
                }
        )
        db[collections[i]].drop();
        sleep(5000);
}

function sleep(milliseconds) {
  var start = new Date().getTime();
  while (new Date().getTime()< start + milliseconds)  {
    // Do nothing
  }
}