Cron Automation

Rift provides automated file processing capabilities through dedicated cron scripts. These scripts enable hands-free processing of dye files, input files, and output files at regular intervals.

Overview

Three cron scripts are available for automated file processing:

  • dye-cron.sh: Automated processing of dye signature files

  • input-cron.sh: Automated processing of input files with atomic operations

  • output-cron.sh: Automated processing of output files with atomic operations

All scripts are designed to run safely in production environments with comprehensive logging, error handling, and concurrent execution prevention.

Features

Common Features

All cron scripts provide:

  • Lock-based execution: Prevents multiple instances from running simultaneously

  • Automatic log rotation: Rotates log files when they exceed 10MB

  • Comprehensive logging: Detailed timestamped logging with automatic file output

  • System health checks: Validates directories and sudo access before processing

  • Signal handling: Graceful cleanup on script termination

  • Error recovery: Robust error handling with detailed error reporting

  • Configuration flexibility: Environment variable-based configuration

Dye File Cron Script

The dye-cron.sh script automatically processes .dye files from a source directory to multiple target directories.

Script Location

  • Development: tools/dye-cron.sh

  • Installed: /usr/share/rift/dye-cron.sh

Configuration

The script uses the following configuration variables:

# Source and target directories
DYE_SOURCE_DIR="/var/abyss/dye"
DYE_TARGET_DIR1="/opt/exports/abyss-default/signatures-local/deep-core-main"
DYE_TARGET_DIR2="/opt/exports/abyss-default/signatures/deep-core-main/current/signatures/local"

# File ownership and permissions
DYE_OWNER_UID=500
DYE_OWNER_GID=500
DYE_PERMISSIONS=644

# User configuration
RIFT_USER="${RIFT_USER:-ec2-user}"

# Logging
LOG_FILE="/var/log/dye-processing.log"
MAX_LOG_SIZE=10485760  # 10MB

Processing Workflow

  1. Lock Acquisition: Prevents concurrent execution

  2. Directory Validation: Ensures all required directories exist

  3. Sudo Verification: Confirms passwordless sudo access

  4. File Discovery: Finds all .dye files in source directory

  5. File Processing: Copies files to both target directories

  6. Permission Setting: Sets proper ownership and permissions

  7. Source Cleanup: Removes source files after successful deployment

  8. Logging: Records all operations with timestamps

Installation and Setup

Step 1: Copy Script to System Location

sudo cp tools/dye-cron.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/dye-cron.sh

Step 2: Configure Passwordless Sudo

Add the following to /etc/sudoers for the ec2-user (or your RIFT_USER):

# For specific commands (recommended)
ec2-user ALL=(ALL) NOPASSWD: /bin/cp, /bin/rm, /bin/chown, /bin/chmod, /usr/bin/find, /usr/bin/stat, /usr/bin/test

# Or for broader access (less secure)
ec2-user ALL=(ALL) NOPASSWD: ALL

Step 3: Set Up Log File

sudo touch /var/log/dye-processing.log
sudo chown ec2-user:ec2-user /var/log/dye-processing.log
sudo chmod 644 /var/log/dye-processing.log

Step 4: Install Cron Job

Switch to the appropriate user and add the cron job:

# Switch to the RIFT_USER (default: ec2-user)
sudo -u ec2-user crontab -e

# Add this line to run every 5 minutes
*/5 * * * * /usr/local/bin/dye-cron.sh >> /var/log/dye-processing.log 2>&1

Alternative Cron Frequencies:

# Every minute (high frequency)
* * * * * /usr/local/bin/dye-cron.sh >> /var/log/dye-processing.log 2>&1

# Every 10 minutes (moderate frequency)
*/10 * * * * /usr/local/bin/dye-cron.sh >> /var/log/dye-processing.log 2>&1

# Every hour (low frequency)
0 * * * * /usr/local/bin/dye-cron.sh >> /var/log/dye-processing.log 2>&1

Input File Cron Script

The input-cron.sh script automatically processes input files from a source directory to a target directory using atomic copy operations.

Script Location

  • Development: tools/input-cron.sh

  • Installed: /usr/share/rift/input-cron.sh

Configuration

The script uses the following configuration variables (all configurable via environment variables):

# Source and target directories
INPUT_SOURCE_DIR="${INPUT_SOURCE_DIR:-/var/abyss/input}"
INPUT_TARGET_DIR="${INPUT_TARGET_DIR:-/data/io-service/input-undersluice-default}"
INPUT_PROCESSED_DIR="${INPUT_PROCESSED_DIR:-${INPUT_SOURCE_DIR}/processed}"

# File ownership and permissions
INPUT_OWNER_UID="${INPUT_OWNER_UID:-500}"
INPUT_OWNER_GID="${INPUT_OWNER_GID:-500}"
INPUT_PERMISSIONS="${INPUT_PERMISSIONS:-644}"

# User configuration
RIFT_USER="${RIFT_USER:-rift}"

# Logging
LOG_FILE="/var/log/input-processing.log"
MAX_LOG_SIZE=10485760  # 10MB

Processing Workflow

  1. Lock Acquisition: Prevents concurrent execution

  2. Directory Validation: Ensures source, target, and processed directories exist

  3. Processed Directory Creation: Auto-creates processed directory if needed

  4. Sudo Verification: Confirms passwordless sudo access

  5. File Discovery: Finds all files in source directory (any file type)

  6. Atomic Processing: Uses temporary files for atomic operations

  7. Permission Setting: Sets proper ownership and permissions

  8. Source File Archival: Moves original files to processed directory after successful copy

  9. Logging: Records all operations with timestamps

Key Differences from Dye Processing

  • Single Target: Copies to one target directory instead of two

  • File Types: Processes all file types, not just .dye files

  • Source Archival: Moves source files to processed directory (dye files are deleted)

  • Reprocessing Prevention: Processed directory prevents duplicate processing

  • Atomic Operations: Uses temporary files and atomic moves for safety

  • Default User: Uses rift user by default instead of ec2-user

Installation and Setup

Step 1: Copy Script to System Location

sudo cp tools/input-cron.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/input-cron.sh

Step 2: Configure Passwordless Sudo

Add the following to /etc/sudoers for the ec2-user (or your RIFT_USER):

# For specific commands (recommended)
ec2-user ALL=(ALL) NOPASSWD: /bin/cp, /bin/mv, /bin/rm, /bin/chown, /bin/chmod, /usr/bin/find, /usr/bin/stat, /usr/bin/test

# Or for broader access (less secure)
ec2-user ALL=(ALL) NOPASSWD: ALL

Step 3: Set Up Log File

sudo touch /var/log/input-processing.log
sudo chown ec2-user:ec2-user /var/log/input-processing.log
sudo chmod 644 /var/log/input-processing.log

Step 4: Install Cron Job

Switch to the ec2-user and add the cron job:

# Switch to the ec2-user
sudo -u ec2-user crontab -e

# Add this line to run every 5 minutes
*/5 * * * * /usr/local/bin/input-cron.sh >> /var/log/input-processing.log 2>&1

Output File Cron Script

The output-cron.sh script automatically processes output files from a source directory to a target directory using atomic copy operations.

Script Location

  • Development: tools/output-cron.sh

  • Installed: /usr/share/rift/output-cron.sh

Configuration

The script uses the following configuration variables (all configurable via environment variables):

# Source and target directories
OUTPUT_SOURCE_DIR="${OUTPUT_SOURCE_DIR:-/opt/exports/abyss-default/outputs/dataExporterinspection}"
OUTPUT_TARGET_DIR="${OUTPUT_TARGET_DIR:-/var/abyss/output}"
OUTPUT_PROCESSED_DIR="${OUTPUT_PROCESSED_DIR:-${OUTPUT_SOURCE_DIR}/processed}"

# File ownership and permissions
OUTPUT_OWNER_UID="${OUTPUT_OWNER_UID:-500}"
OUTPUT_OWNER_GID="${OUTPUT_OWNER_GID:-500}"
OUTPUT_PERMISSIONS="${OUTPUT_PERMISSIONS:-644}"

# User configuration
RIFT_USER="${RIFT_USER:-rift}"

# File expiration configuration (IMPORTANT)
FILE_EXPIRATION_HOURS="${FILE_EXPIRATION_HOURS:-24}"

# Logging
LOG_FILE="/var/log/output-processing.log"
MAX_LOG_SIZE=10485760  # 10MB

Warning

File Expiration Policy: The output cron script automatically deletes files older than 24 hours (configurable via FILE_EXPIRATION_HOURS) from:

  • /var/abyss/output (target directory)

  • /var/abyss/input/processed (processed input files)

This is done to save disk space. Files are permanently deleted and cannot be recovered. If you need to retain files longer than 24 hours, set FILE_EXPIRATION_HOURS to a higher value or back up important files before they expire.

Processing Workflow

  1. Lock Acquisition: Prevents concurrent execution

  2. Directory Validation: Ensures source, target, and processed directories exist

  3. Processed Directory Creation: Auto-creates processed directory if needed

  4. Sudo Verification: Confirms passwordless sudo access

  5. File Discovery: Finds all files in source directory (any file type)

  6. Atomic Processing: Uses temporary files for atomic operations

  7. Permission Setting: Sets proper ownership and permissions

  8. Source File Archival: Moves original files to processed directory after successful copy

  9. File Expiration Cleanup: Deletes files older than configured threshold (default 24 hours) from target and processed directories

  10. Logging: Records all operations with timestamps

Key Features

  • Single Target: Copies to one target directory

  • File Types: Processes all file types, not just specific extensions

  • Source Archival: Moves source files to processed directory to prevent reprocessing

  • Atomic Operations: Uses temporary files and atomic moves for safety

  • Default User: Uses rift user by default

  • System Output: Handles system-generated output files for downstream consumption

Installation and Setup

Step 1: Copy Script to System Location

sudo cp tools/output-cron.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/output-cron.sh

Step 2: Configure Passwordless Sudo

Add the following to /etc/sudoers for the rift user (or your RIFT_USER):

# For specific commands (recommended)
rift ALL=(ALL) NOPASSWD: /bin/cp, /bin/mv, /bin/rm, /bin/chown, /bin/chmod, /usr/bin/find, /usr/bin/stat, /usr/bin/test

# Or for broader access (less secure)
rift ALL=(ALL) NOPASSWD: ALL

Step 3: Set Up Log File

sudo touch /var/log/output-processing.log
sudo chown rift:rift /var/log/output-processing.log
sudo chmod 644 /var/log/output-processing.log

Step 4: Install Cron Job

Switch to the rift user and add the cron job:

# Switch to the rift user
sudo -u rift crontab -e

# Add this line to run every 5 minutes
*/5 * * * * /usr/local/bin/output-cron.sh >> /var/log/output-processing.log 2>&1

Custom Configuration

Environment Variable Override

All scripts support environment variable customization:

Dye Cron Script:

# Custom user
export RIFT_USER=myuser

# Then install cron job as that user
sudo -u myuser crontab -e

Input Cron Script:

# Custom directories
export INPUT_SOURCE_DIR=/custom/source
export INPUT_TARGET_DIR=/custom/target

# Custom ownership
export INPUT_OWNER_UID=1000
export INPUT_OWNER_GID=1000

# Custom permissions
export INPUT_PERMISSIONS=755

# Custom user
export RIFT_USER=myuser

Output Cron Script:

# Custom directories
export OUTPUT_SOURCE_DIR=/custom/source
export OUTPUT_TARGET_DIR=/custom/target

# Custom ownership
export OUTPUT_OWNER_UID=1000
export OUTPUT_OWNER_GID=1000

# Custom permissions
export OUTPUT_PERMISSIONS=755

# Custom file expiration (hours)
export FILE_EXPIRATION_HOURS=48  # Keep files for 48 hours instead of default 24

# Custom user
export RIFT_USER=myuser

To use custom environment variables in cron, add them to the crontab:

# Edit crontab
sudo -u ec2-user crontab -e

# Add environment variables at the top
INPUT_SOURCE_DIR=/custom/source
INPUT_TARGET_DIR=/custom/target
FILE_EXPIRATION_HOURS=48
RIFT_USER=myuser

# Then add the cron jobs
*/5 * * * * /usr/local/bin/input-cron.sh >> /var/log/input-processing.log 2>&1
*/5 * * * * /usr/local/bin/output-cron.sh >> /var/log/output-processing.log 2>&1

Monitoring and Management

Checking Cron Job Status

View Current Cron Jobs:

# For dye processing (ec2-user)
sudo -u ec2-user crontab -l

# For input processing (ec2-user)
sudo -u ec2-user crontab -l

Check Running Processes:

# Check for running cron scripts
ps aux | grep -E "(dye-cron|input-cron|output-cron)"

# Check PID files
cat ${TMPDIR:-/tmp}/rift-cron/dye-cron.pid 2>/dev/null
cat ${TMPDIR:-/tmp}/rift-cron/input-cron.pid 2>/dev/null
cat ${TMPDIR:-/tmp}/rift-cron/output-cron.pid 2>/dev/null

Check Lock Files:

# Check for active locks
ls -la ${TMPDIR:-/tmp}/rift-cron/*-cron.lock 2>/dev/null

Log Monitoring

Monitor Real-time Processing:

# Dye file processing
tail -f /var/log/dye-processing.log

# Input file processing
tail -f /var/log/input-processing.log

View Recent Activity:

# Today's dye processing activity
grep "$(date '+%Y-%m-%d')" /var/log/dye-processing.log

# Today's input processing activity
grep "$(date '+%Y-%m-%d')" /var/log/input-processing.log

Check Log File Sizes:

ls -lh /var/log/*-processing.log*

Manual Testing

Test the cron scripts manually before installing them:

# Test dye cron script
/usr/local/bin/dye-cron.sh

# Test input cron script
/usr/local/bin/input-cron.sh

# Test with custom environment
RIFT_USER=testuser /usr/local/bin/input-cron.sh

Troubleshooting

Common Issues

Cron Job Not Running

  1. Check crontab installation:

    sudo -u ec2-user crontab -l  # or -u rift

  2. Verify script permissions:

    ls -la /usr/local/bin/*-cron.sh

  3. Check system cron service:

    sudo systemctl status crond

Permission Denied Errors

  1. Verify passwordless sudo:

    sudo -u ec2-user sudo -n true && echo "OK" || echo "FAILED"

  2. Check sudoers configuration:

    sudo visudo -c  # Check syntax
sudo grep ec2-user /etc/sudoers

  3. Test manual execution:

    sudo -u ec2-user /usr/local/bin/dye-cron.sh

Lock File Issues

Lock files are now stored in a user-writable directory (${TMPDIR:-/tmp}/rift-cron/) to avoid permission issues. The scripts automatically create this directory if it doesn’t exist.

  1. Remove stale locks:

    rm -f ${TMPDIR:-/tmp}/rift-cron/*-cron.lock ${TMPDIR:-/tmp}/rift-cron/*-cron.pid

  2. Check for zombie processes:

    ps aux | grep -E "(dye-cron|input-cron|output-cron)" | grep -v grep

Directory Not Found Errors

  1. Create missing directories:

    sudo mkdir -p /var/abyss/dye
sudo mkdir -p /var/abyss/input
sudo mkdir -p /var/abyss/output

  2. Check directory permissions:

    ls -la /var/abyss/

Log File Issues

  1. Check log directory permissions:

    ls -la /var/log/ | grep processing

  2. Create log files manually:

    sudo touch /var/log/dye-processing.log
sudo chown ec2-user:ec2-user /var/log/dye-processing.log

Debug Mode

Both scripts provide detailed logging. To increase verbosity, check the log files:

# Watch logs in real-time
tail -f /var/log/dye-processing.log
tail -f /var/log/input-processing.log

Performance Monitoring

Monitor Processing Statistics:

# Count processed files today
grep "$(date '+%Y-%m-%d')" /var/log/dye-processing.log | grep "Processing file" | wc -l

# Check average processing time
grep "Processing completed" /var/log/dye-processing.log | tail -10

Monitor System Resources:

# Check disk space
df -h /var/log/
df -h /var/abyss/

# Check system load during processing
top -p $(pgrep -f "cron.sh")

Security Considerations

Sudo Configuration

  • Minimal Privileges: Use specific command restrictions instead of NOPASSWD: ALL

  • User Isolation: Use dedicated users for different cron scripts

  • Regular Audits: Review sudoers configuration regularly

File Permissions

  • Log Files: Ensure log files are not world-readable if they contain sensitive information

  • Script Files: Ensure cron scripts are not writable by unauthorized users

  • Lock Files: Verify lock files are created with proper permissions in user-writable directory

Network Security

  • File Transfer: If processing files from network sources, ensure secure transfer protocols

  • Access Control: Implement proper access controls on source and target directories

Best Practices

  1. Start with Manual Testing

    Always test cron scripts manually before installing them in cron.

  2. Use Appropriate Frequencies

    • High-volume environments: Every 1-5 minutes

    • Normal environments: Every 5-15 minutes

    • Low-volume environments: Every 30-60 minutes

  3. Monitor Log Files

    Set up log monitoring and alerting for error conditions.

  4. Regular Maintenance

    • Review log files regularly

    • Clean up old log files

    • Monitor disk space usage

  5. Environment Consistency

    Use the same environment variables and configurations across development and production.

  6. Backup Considerations

    • Consider backing up source directories before processing

    • Implement file retention policies for processed files

Integration with Rift Commands

The cron scripts complement the manual Rift commands:

  • Manual processing: Use rift dye-add and rift input-add for immediate processing

  • Automated processing: Use cron scripts for continuous, hands-free operation

  • Monitoring: Use both manual commands and log files for status checking

See Dye File Management and Input File Management for manual command documentation.