Clearing the taskflow_execution_log Collection

The taskflow_execution_log collection contains records related to the logs generated during the execution of actions in the respective Integration Task Flow steps.

Record Cleaning Script:

The following script is used to delete records from the taskflow_execution_log collection in MongoDB that are older than the specified hour.

This operation helps optimize storage space by cleaning up the database.

#!/bin/bash

# Information required to connect to the MongoDB server
HOST="<MONGODB_MASTER_IP_ADDRESS>"
PORT="<MONGODB_PORT>"
DB_NAME="apinizerdb"
USERNAME="apinizer"
PASSWORD="<PASSWORD>"
AUTH_DB="admin"
TARGET_COLLECTION="taskflow_execution_log"
TARGET_LOG_LOCATION=$(pwd)/purge_taskflow_execution_log_collection.log

echo "Script started on $(date +"%Y-%m-%d %H:%M:%S")" >> $TARGET_LOG_LOCATION  

# Variable specifying how many hours/days ago documents will be deleted
TIME_VALUE="1D"  #You can change this value as desired.
 
# Analyzing and converting the TIME_VALUE value to milliseconds.
if [[ $TIME_VALUE =~ [Hh]$ ]]; then
    HOURS_AGO=${TIME_VALUE%H*}
    TIME_IN_MILLISECONDS=$(($HOURS_AGO * 60 * 60 * 1000))
elif [[ $TIME_VALUE =~ [Dd]$ ]]; then
    DAYS_AGO=${TIME_VALUE%D*}
    TIME_IN_MILLISECONDS=$(($DAYS_AGO * 24 * 60 * 60 * 1000))
else
    echo "Invalid time format. Please enter a value ending with H (hour) or D (day)." >> $TARGET_LOG_LOCATION
    exit 1
fi

MONGO_COMMANDS=$(cat <<EOF
var dateToRemove=new Date((new Date().getTime() - $TIME_IN_MILLISECONDS));
var bulk = db.getCollection("$TARGET_COLLECTION").initializeUnorderedBulkOp();
bulk.find( {"auditEventDate":{"\$lt": dateToRemove}}).remove();
bulk.execute();
EOF
)

# Execute the command and write its output to the log file.
{
    mongosh mongodb://$USERNAME:$PASSWORD@$HOST:$PORT/$DB_NAME --authenticationDatabase $AUTH_DB --eval "$MONGO_COMMANDS"
    # Check the completion of the command.
    if [ $? -ne 0 ]; then
        echo "MongoDB command failed." >> $TARGET_LOG_LOCATION
        exit 1
    fi
} >> $TARGET_LOG_LOCATION 2>&1

echo "The command has been executed. Please check the content of the related collection." >> $TARGET_LOG_LOCATION

echo "Script finished on $(date +"%Y-%m-%d %H:%M:%S")" >> $TARGET_LOG_LOCATION

CODE

The time filter is applied based on UTC.

How It Works:

The required information to connect to the MongoDB server is defined in the relevant variables: HOST, PORT, DB_NAME, USERNAME, PASSWORD, and AUTH_DB.
The variable TIME_VALUE specifies how many hours ago records will be deleted. This value can be in hours (H) or days (D).
Within the MONGO_COMMANDS variable, MongoDB commands are defined to find and delete documents older than the specified number of hours.
The MongoDB server is connected using the mongosh command, and the commands defined in the MONGO_COMMANDS variable are executed.
When the process is completed, the message "The command has been executed. Please check the content of the related collection." is printed to the screen.

Usage:

Before running the script, enter your own information into the MongoDB variables, and update the TIME_VALUE variable according to the desired hour/day value. For example, set TIME_VALUE=3H to keep documents from the last 3 hours and delete the rest, or set TIME_VALUE=5D to keep documents from the last 5 days and delete the rest.
To run the script on a Linux-based operating system, first copy the script to a file using a text editor (vi, nano, etc.) and edit the variables inside.
Execute permission is granted to the file: chmod +x purge_taskflow_execution_log_collection.sh
The file is executed.: ./purge_taskflow_execution_log_collection.sh
- If you expect the process to take a long time, you can add a space and the & character at the end of the command to make it run in the background.: ./purge_taskflow_execution_log_collection.sh &

chmod +x /path/to/purge_taskflow_execution_log_collection.sh
./path/to/purge_taskflow_execution_log_collection.sh &

CODE

This process can be done manually or set to repeat at specified intervals. To repeat it, you need to add this task to Linux cronjob settings.

CronJob Usage:

1) Open the cron editor by running the following command in the terminal:

crontab -e

CODE

2) In the opened editor, add a line based on how frequently you want to run the script.

For example, to run it every day at 23:59, you can write:

59 23 * * * /path/to/purge_taskflow_execution_log_collection.sh > /path/to/logfile.log 2>&1

CODE

To save the added line, press the Esc key, type :wq, and press the Enter key.

Reminder

The data clearing process in the script will occur according to the time zone specified within the script.

Therefore, it is important to ensure consistency between the time zone in the script and the triggering time set with cron.