Clearing Collections that Store Trace Logs

When Trace is enabled, the collections named log_apiproxytraffic, log_apiproxytraffic_policy_execution, and log_apiproxytraffic_routing record details of API traffic. Even after Trace is turned off, these data continue to be stored in the database. These records allow for the examination of the state before and after the application of policies in the relevant proxy, as well as traffic and routing information, on the Trace screen.

Record Cleaning Script:

The script below is used to delete records from the log_apiproxytraffic, log_apiproxytraffic_policy_execution, and log_apiproxytraffic_routing collections in MongoDB for application logs stored in MongoDB up to a specified hour.

This operation helps optimize storage space by cleaning up the database.

#!/bin/bash

# Information required to connect to the MongoDB server
HOST="<MONGODB_MASTER_IP_ADDRESS>"
PORT="<MONGODB_PORT>"
DB_NAME="apinizerdb"
USERNAME="apinizer"
PASSWORD="<PASSWORD>"
AUTH_DB="admin"
TARGET_COLLECTIONS=("log_apiproxytraffic" "log_apiproxytraffic_policy_execution" "log_apiproxytraffic_routing")
TARGET_LOG_LOCATION=$(pwd)/purge_trace_logs.log

echo "Script started on $(date +"%Y-%m-%d %H:%M:%S")" >> $TARGET_LOG_LOCATION  

# Variable specifying how many hours/days ago documents will be deleted
TIME_VALUE="1D"  #You can change this value as desired.

QUERY_CONDITION=""

# Analyzing and converting the TIME_VALUE value to milliseconds.
if [[ ! -z $TIME_VALUE ]]; then
    if [[ $TIME_VALUE =~ [Hh]$ ]]; then
        HOURS_AGO=${TIME_VALUE%H*}
        TIME_IN_MILLISECONDS=$(($HOURS_AGO * 60 * 60 * 1000))
        QUERY_CONDITION="\"timestamp\": {\"\$lt\": new Date((new Date().getTime() - $TIME_IN_MILLISECONDS))}"
    elif [[ $TIME_VALUE =~ [Dd]$ ]]; then
        DAYS_AGO=${TIME_VALUE%D*}
        TIME_IN_MILLISECONDS=$(($DAYS_AGO * 24 * 60 * 60 * 1000))
        QUERY_CONDITION="\"timestamp\": {\"\$lt\": new Date((new Date().getTime() - $TIME_IN_MILLISECONDS))}"
    else             
		echo "Invalid time format. Please enter a value ending with H (hour) or D (day)." >> $TARGET_LOG_LOCATION
         exit 1
    fi
fi

if [[ -z $QUERY_CONDITION ]]; then
    echo "TIME_VALUE is not valid." >> $TARGET_LOG_LOCATION
    exit 1
fi

for COLLECTION in "${TARGET_COLLECTIONS[@]}"; do
    MONGO_COMMANDS=$(cat <<EOF
    var bulk = db.getCollection("$COLLECTION").initializeUnorderedBulkOp();
    bulk.find({$QUERY_CONDITION}).remove();
    bulk.execute();
	EOF
    )

	# Execute the command and write its output to the log file.
{
        mongosh mongodb://$USERNAME:$PASSWORD@$HOST:$PORT/$DB_NAME --authenticationDatabase $AUTH_DB --eval "$MONGO_COMMANDS"
        # Check the completion of the command.
		if [ $? -ne 0 ]; then
            echo "MongoDB command for collection $COLLECTION failed." >> $TARGET_LOG_LOCATION
            exit 1
        fi
    } >> $TARGET_LOG_LOCATION 2>&1
done  

echo "The command has been executed. Please check the content of the related collection." >> $TARGET_LOG_LOCATION
 
echo "Script finished on $(date +"%Y-%m-%d %H:%M:%S")" >> $TARGET_LOG_LOCATION

CODE

The time filter is applied based on UTC.

How It Works:

The required information to connect to the MongoDB server is defined in the relevant variables: HOST, PORT, DB_NAME, USERNAME, PASSWORD, and AUTH_DB.
The variable TIME_VALUE specifies how many hours ago records will be deleted. This value can be in hours (H) or days (D).
Within the MONGO_COMMANDS variable, MongoDB commands are defined to find and delete documents older than the specified number of hours.
The MongoDB server is connected using the mongosh command, and the commands defined in the MONGO_COMMANDS variable are executed.
When the process is completed, the message "The command has been executed. Please check the content of the related collection." is printed to the screen.

Usage:

Before running the script, enter your own information into the MongoDB variables, and update the TIME_VALUE variable according to the desired hour/day value. For example, set TIME_VALUE=3H to keep documents from the last 3 hours and delete the rest, or set TIME_VALUE=5D to keep documents from the last 5 days and delete the rest.
To run the script on a Linux-based operating system, first copy the script to a file using a text editor (vi, nano, etc.) and edit the variables inside.
Execute permission is granted to the file: chmod +x purge_trace_logs.sh
The file is executed.: ./purge_trace_logs.sh
- If you expect the process to take a long time, you can add a space and the & character at the end of the command to make it run in the background.: ./purge_trace_logs.sh &

chmod +x /path/to/purge_trace_logs.sh
./path/to/purge_trace_logs.sh &

CODE

This process can be done manually or set to repeat at specified intervals. To repeat it, you need to add this task to Linux cronjob settings.

CronJob Usage:

1) Open the cron editor by running the following command in the terminal:

crontab -e

CODE

2) In the opened editor, add a line based on how frequently you want to run the script.

For example, to run it every day at 23:59, you can write:

59 23 * * * /path/to/purge_trace_logs.sh > /path/to/logfile.log 2>&1

CODE

To save the added line, press the Esc key, type :wq, and press the Enter key.

Reminder

The data clearing process in the script will occur according to the time zone specified within the script.

Therefore, it is important to ensure consistency between the time zone in the script and the triggering time set with cron.