The Bill Murray quote on Daniel Wayne Armstrong’s site is a good one, really makes you think. The site is great, lots of useful content about Linux and the Raspberry Pi, plus OpenWRT router mods. I have learned a lot ( and used it ) by skimming through his posts. He also has some interesting quotes like the Bill Murray one below.
Bill Murray on staying relaxed and how success will follow:
http://www.circuidipity.com/20140918.html
In January 2015 I created a post about automatically creating a status page for a Linux server that I have. Typically this is put under a restricted directory and allows you to see a snapshot of what is happening with the server. I run it by putting the scripts in the /etc/cron.hourly directory on a Linux PC and a Raspberry Pi running Linux.
It serves as a simple way to check up on the server without having to use a tool such as Webmin that requires a login. It also keeps a trail of log files that get rotated on a monthly basis, so there is always a few old ones around to track down any problems and patterns in the operation.
I have found this information useful when I have traced down malfunctions that can occur when setting up a server and also when I was trying to get a webcam up and running and had the USB bus hang up a few times when the cam was overloaded with too much light.
In the new script file I fixed a bug by adding parenthesis around a line that I was trying to echo and I added code to run the w command to show a quick picture on who is logged in, how long the server has been up and running and the values for the average load on the server at the 1, 5 and 15 minute marks.
#!/bin/dash # Remove old log rm /var/www/status/log.txt # Print logged outputs into new log.txt date >> /var/www/status/log.txt echo >> /var/www/status/log.txt tail /var/log/syslog >> /var/www/status/log.txt echo >> /var/www/status/log.txt free >> /var/www/status/log.txt echo >> /var/www/status/log.txt df -h >> /var/www/status/log.txt echo >> /var/www/status/log.txt # Top memory using processes http://www.commandlinefu.com/commands/view/3/display-the-top-ten-running-processes-sorted-by-memory-usage #ps aux | sort -nk +4 | tail >> log.txt echo "USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND" >> /var/www/status/log.txt ps aux | sort -nrk 4 | head >> /var/www/status/log.txt echo >> /var/www/status/log.txt # Logged in User info using w command w >> /var/www/status/log.txt echo >> /var/www/status/log.txt echo >> /var/www/status/log.txt # Copy log.txt into the full log that is collected cat /var/www/status/log.txt >> /var/www/status/fulllog.txt # Create a free standind copy of the process tree pstree > /var/www/status/pstree.txt
I also created a version of the script for a desktop Linux PC that does not have Apache installed. In it I use a DIR variable to contain the directory that I want the log.txt file stored.
#!/bin/dash # User defined variables # No trailing / on DIR! DIR=/home/erick/status # Remove old log rm $DIR/log.txt # Print logged outputs into new log.txt date >> $DIR/log.txt echo >> $DIR/log.txt tail /var/log/syslog >> $DIR/log.txt echo >> $DIR/log.txt free >> $DIR/log.txt echo >> $DIR/log.txt df -h >> $DIR/log.txt echo >> $DIR/log.txt # Top memory using processes http://www.commandlinefu.com/commands/view/3/display-the-top-ten-running-processes-sorted-by-memory-usage #ps aux | sort -nk +4 | tail >> log.txt echo "USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND" >> $DIR/log.txt ps aux | sort -nrk 4 | head >> $DIR/log.txt echo >> $DIR/log.txt # Logged in User info using w command w >> $DIR/log.txt echo >> $DIR/log.txt echo >> $DIR/log.txt # Copy log.txt into the full log that is collected cat $DIR/log.txt >> $DIR/fulllog.txt # Create a free standing copy of the process tree pstree > $DIR/pstree.txt
In the /etc/cron.monthly directory I have created a file that is called status-log-rotate and it will save backup copies of 2 months worth of the full concatenated server status logs.
#! /bin/bash DIR=/home/erick/status mv $DIR/fulllog.txt.1 $DIR/fulllog.txt.2 mv $DIR/fulllog.txt $DIR/fulllog.txt.1
For the Raspberry Pi which has an SD card that I am trying to be conscious of writing to often. I have recently made some modifications to put the /tmp folder onto RAM using tmpfs. I create the hourly log underneath a folder there. Daily via a script it cron.hourly it gets concatenated into a daily log which is under a status folder that has restricted access. This gets appended once per day to the fulllog which actually lives on the SD card. The end result, no multiple hourly writes to the log file, just one append to the full log per day. The only downside is if the power drops and then some log entries will be lost for the day.
#!/bin/dash # Set DIR, on Pi this is a temp location for log DIR=/tmp/web # Set fixed DIR FIXDIR for files that have to be stored on SD card # Nevermind, just make a daily log and then copy that to the full log daily. #FIXDIR=/var/www/status # Remove old log rm $DIR/log.txt # Print logged outputs into new log.txt date >> $DIR/log.txt echo >> $DIR/log.txt tail /var/log/syslog >> $DIR/log.txt echo >> $DIR/log.txt free >> $DIR/log.txt echo >> $DIR/log.txt df -h >> $DIR/log.txt echo >> $DIR/log.txt # Top memory using processes http://www.commandlinefu.com/commands/view/3/display-the-top-ten-running-processes-sorted-by-memory-usage echo "USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND" >> $DIR/log.txt ps aux | sort -nrk 4 | head >> $DIR/log.txt echo >> $DIR/log.txt # Logged in User info using w command w >> $DIR/log.txt echo >> $DIR/log.txt echo >> $DIR/log.txt # Copy log.txt into the full log that is collected cat $DIR/log.txt >> $DIR/dailylog.txt # Create a free standing copy of the process tree pstree > $DIR/pstree.txt
#! /bin/bash DIR=/tmp/web FIXDIR=/var/www/status echo "----------------------------------------------" >> $DIR/dailylog.txt date >> $DIR/dailylog.txt echo "----------------------------------------------" >> $DIR/dailylog.txt cat $DIR/dailylog.txt >> $FIXDIR/fulllog.txt rm $DIR/dailylog.txt
Below is what the logcreate script will output to the log.txt file on a Raspberry Pi that I have running as a web server.
Sun Jul 12 14:17:01 EDT 2015
Jul 12 13:47:51 raspberrypi dhclient: DHCPACK from 192.168.1.1
Jul 12 13:47:52 raspberrypi dhclient: bound to 192.168.1.17 -- renewal in 40673 seconds.
Jul 12 13:59:01 raspberrypi /USR/SBIN/CRON[28010]: (erick) CMD (aplay /opt/sonic-pi/etc/samples/guit_e_fifths.wav)
Jul 12 13:59:07 raspberrypi /USR/SBIN/CRON[28009]: (CRON) info (No MTA installed, discarding output)
Jul 12 14:00:01 raspberrypi /USR/SBIN/CRON[28013]: (erick) CMD (/home/erick/fswebcam/cron-timelapse.sh >> timelapse.log)
Jul 12 14:00:23 raspberrypi /USR/SBIN/CRON[28012]: (CRON) info (No MTA installed, discarding output)
Jul 12 14:01:01 raspberrypi /USR/SBIN/CRON[28022]: (root) CMD (/home/erick/bin/usbreset /dev/bus/usb/001/004)
Jul 12 14:01:02 raspberrypi /USR/SBIN/CRON[28021]: (CRON) info (No MTA installed, discarding output)
Jul 12 14:09:01 raspberrypi /USR/SBIN/CRON[28053]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Jul 12 14:17:01 raspberrypi /USR/SBIN/CRON[28064]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
total used free shared buffers cached
Mem: 445804 424488 21316 0 106768 260516
-/+ buffers/cache: 57204 388600
Swap: 102396 0 102396
Filesystem Size Used Avail Use% Mounted on
rootfs 6.3G 3.1G 3.0G 51% /
/dev/root 6.3G 3.1G 3.0G 51% /
devtmpfs 214M 0 214M 0% /dev
tmpfs 44M 240K 44M 1% /run
tmpfs 5.0M 8.0K 5.0M 1% /run/lock
tmpfs 88M 0 88M 0% /run/shm
/dev/mmcblk0p5 60M 19M 41M 32% /boot
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2071 0.0 3.0 24896 13652 ? Ss Jun28 2:24 /usr/sbin/apache2 -k start
www-data 27745 0.0 1.5 25412 7084 ? S 09:58 0:00 /usr/sbin/apache2 -k start
www-data 27744 0.0 1.5 24960 6760 ? S 09:58 0:00 /usr/sbin/apache2 -k start
www-data 27743 0.0 1.5 25428 7116 ? S 09:58 0:00 /usr/sbin/apache2 -k start
www-data 27742 0.0 1.5 25396 7036 ? S 09:58 0:00 /usr/sbin/apache2 -k start
www-data 27538 0.0 1.5 25396 7032 ? S 06:25 0:00 /usr/sbin/apache2 -k start
www-data 27502 0.0 1.5 25404 7036 ? S 06:25 0:00 /usr/sbin/apache2 -k start
www-data 27501 0.0 1.5 25396 7044 ? S 06:25 0:00 /usr/sbin/apache2 -k start
www-data 27747 0.0 1.3 24936 6188 ? S 09:58 0:00 /usr/sbin/apache2 -k start
www-data 27746 0.0 1.3 24936 6188 ? S 09:58 0:00 /usr/sbin/apache2 -k start
14:17:02 up 14 days, 12:56, 1 user, load average: 0.00, 0.01, 0.05
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
erick pts/0 192.168.1.5 14:04 10:39 1.70s 1.70s -bash
I had a Bonsai tree that was neglected at times over the winter when we were away from home. It was not getting enough water, dried out and lost all of it’s leaves. I had a few other plants die off but most of them are easily replaced from commonly obtainable seeds. This Bonsai however was a plant that I had for about 10 years and it was disappointing to lose it.
In early May, I pulled it out of it’s pot and put it on a pile to be discarded. Luckily the place where it was put was in the shade. Plus this year (2015) we have received an exceptional amount of rain through June, more than double the average. All of this helped to keep it alive at some low level, even without soil.
In mid-July, it started to get a few little green leaves on it. What a miracle, it had some life left in it after all! Now it is back in the pot and I will eventually see how many branches are alive.
This is the second plant that has come back from the dead this year, the Bonsai along with a Ginko Biloba Tree that came back from the dead.
Frequently when using the non command line version of PHPList which I have loaded on my service providers server, I have to hit Process Queue multiple times for a message. I have read a lot of posts online complaining this issue. I have done some troubleshooting of it myself by turning on the verbose feature in PHPList and I am still at a loss as to what it gets stuck on and then stops processing the queue.
PHPList works great, I love it for all that it does and does well. My one complaint is the Process Queue issue and from reading a bunch of posts online I have successfully patched together a solution.
This code can be executed on the server where PHPList lives if they will let you run a wget command from a CRON entry, some won’t. In my case I opted to simply run the command from a Raspberry Pi server that I have that runs 24/7. It is simple a looping script that does the equivalent of hitting the “Process Queue” button on PHPList at regularly timed intervals and detected when the queue has been processed. I have seen several examples of using wget to be fired off every 5 minutes to do this, but that means it gets run every 5 minutes, all day, every day. Why not run something that can be scheduled for a time and then runs to completion which this script does.
This script also produces a log file that can be copied to a location viewable via the server so you can double check to see if all ran OK. I see this mostly as a debug feature and it could potentially be turned off along with some of the “echos” of text. Except for the one in the else-if, it is an error to have an empty else-if in a bash script.
#!/bin/bash # Controller Script for PHPlist Process Queue # Erick Clasen blog.oils-of-life.com 05/16/2015
# User defined variables # No trailing / on DIR! DIR=/home/erick/php-proc LOG=php-pq.log # Time to sleep between loop iterations. TIME=3m # Max number of loops, so the script breaks out if for some reason it fails to complete. LOOPS=100 # Publish a copy of the log to the following directory PUBDIR=/home/erick/public_html
# wget having trouble with some options so simply switch to the dir we want it to work in. cd $DIR
x=1 echo Start------------- >> $LOG
# While loop, breaks out if done or at LOOPS as a guard against #infinite loop. while [ $x -le $LOOPS ]; do
# Timestamp in DDMMYYYY_HHMM-SS (UTC) timestamp=$(date -u +"%d%m%Y_%H%M-%S")
echo $x $timestamp echo $x $timestamp >> $LOG
x=$(( $x + 1 ))
# Use wget to run process queue with login and password provided wget 'your_url_here.com/phpList/admin/?page=processqueue&login=phplist&password=your_password_here'
# Test to see if PHPlist process queue has completed. if grep -q "Finished, Nothing to do" $DIR/index.* then # Exit from Loop with a break, mark log,remove index files that wget fetched, break out. echo All Done! Breaking Out echo All Done!--------- >> $LOG echo ------------------ >> $LOG # Publish Log, Optional cp $LOG $PUBDIR/$LOG rm $DIR/index.* break else # Nothing Happens echo Keep Running... fi
# Sleep for TIME echo ------- Sleeping Process --------
sleep $TIME;
done;
Process Bounces can be run from a script as well…
#!/bin/bash # Controller Script for PHPlist Process Bounces # Erick Clasen blog.oils-of-life.com 05/16/2015
# User defined variables # No trailing / on DIR! DIR=/home/erick/php-proc LOG=php-pq.log
# Publish a copy of the log to the following directory PUBDIR=/home/erick/public_html
# wget having trouble with some options so simply switch to the dir we want it $ cd $DIR
# Use wget to run process queue with login and password provided wget 'your_url_here.com/phpList/admin/?page=processbounces&login=phplist&password=your_password_here' echo ---------------------------- >> $LOG echo BOUNCES HAVE BEEN PROCESSED! >> $LOG echo ---------------------------- >> $LOG # Publish Log, Optional cp $LOG $PUBDIR/$LOG rm $DIR/index.*
These entries run Process Queue every Monday, Wednesday and Friday (1,3,5) at 4:15AM and Process Bounces at 12:15AM on the first day of the month. Nothing magic about those settings, you can set them any way you wish. I figure starting the process in the wee hours of the morning will allow people to receive the email by the time that they wake and first check their mail. I set the time to 15 after so it won’t run at the same time as some processes that I have set to run on the hour. Sending mail on weekdays keeps people from having to deal with more email than they might like on the weekends.
15 04 * * 1,3,5 /home/erick/php-proc/php-pq.sh 15 00 1 * * /home/erick/php-proc/php-pb.sh
If you must make PHPList process the queue now and don’t want to wait for the CRON task. Two options are available, do it the via the normal way from the PHPList admin website or you can use something as simple as a CGI script to force the php-pq.sh script to execute. Below is an example of a dirt simple CGI script. It prints out the tail of the php-pq.log and runs the php-pq.sh script. Php-pq.log is the log of the previous automatic processing event and then goes ahead and fires off the Process Queue Override by running the script. Basically the act of loading the page makes it run. I keep my cgi-bin directory locked down so no one outside of a few IP addresses can get to the files there. But it would be fairly harmless if someone did, the worst that happens is that the queue gets processed.
#!/bin/bash echo "Content-type: text/html" echo "" echo "PHPList Automatic Process Queue Override" echo "" echo "
PHPList Automatic Process Queue Override from host $(hostname -s &)
" echo "" echo "
$(tail /home/erick/php-proc/php-pq.log &)
" echo "" echo "
Information generated on $(date &)
" echo "" echo "Processing..."
echo "
$(/home/erick/php-proc/php-pq.sh &)
"
echo "
"
I am working on taking some notes on some changes I have made to the PHPList config.php file and will be posting on this in August 2015.