Category Archives: Web Hosting

Penguin

Pastepile

Future development depends on the determination of future generations to contribute towards building their own society.

Pastepile is a fork of guestbook.cgi – a simple guestbook script, written by John Callender in 1999. It is the last part of his Beginner’s Guide to CGI Scripting with Perl, Running a Guestbook.
This new version, a.k.a Pastepile, is a dirt simple Perl CGI program that creates a blog like pasting tool to be able to paste notes online or on a local server. Allows grabbing snippets of text or code for later use on whether on the local machine or another on the network. I made it sometime after studying John Callender’s tutorial when I was digging into learning about CGI and Perl several years ago.

History

Pastepile originated as a way to paste snippets of code and write short notes on configuration changes that I make on servers, including a Raspberry Pi that I run. The only requirement is the installation of a web server such as Apache. This little Pastepile tool made it easy to keep track of information related to the servers in one place  and search-able via a browser. After using it in the is mode for a while, I cloned it and started to use it as a general place to paste info and write short notes. This makes it easy to save something while on one machine on the network and be able to retrieve it later on that machine or another.

Example of What it Looks Like


Subject: Running a form-to-email gateway at 19:04:44, Sat Feb 3, 2018 from 192.168.1.7

http://www.resoo.org/docs/perl/begperl/form_to_email.html


Subject: Slackware Linux Essentials at 12:10:03, Thu Feb 1, 2018 from 192.168.1.179

www.slackbook.org/html/index.html


Additions:Remote Address,Time,Reversed Order

Besides stripping down guestbook.cgi to remove code specifc to it’s guestbook function and cleaning up the code to reflect it’s new use, three functional code changes were made. One is to add a timestamp to keep track of when the entry was created. Another change made was showing the IP of the machine the post originates from by reading the environment variable $REMOTE_ADDR to keep track of where the post originates from. Many machines on my network have static IP’s so this is handy for me at least. The final change was to reverse the order of the listing. In guestbook.cgi a First In On Top ordering is the layout of the guestbook posts. For Pastepile I reversed it so the most recent and not the oldest post is on top of the list, Last In On Top ordering. This made sense as the most recently written notes are more likely to be important and I want those to be close to the top of the pile and let the file get long with the aging information at the bottom.

/p/

For ease of access via a shorter link a simple index.html redirector page was place in /var/www/p/, following a 4chan-esqe style of folder naming and a nice short link to get to the pastepile.cgi script, which lives in the /usr/lib/cgi-bin/ directory. The /p/ directory also holds the pastepile.html file that is created by pastepile.cgi as it’s data page.

 Redirector Example for /var/www/p/index.html

<HTML>
<HEAD>
<meta http-equiv="refresh" content="0; url='http://192.168.1.17/cgi-bin/pastepile.cgi'"/>
</HEAD>
<BODY>
<p><a href="http://192.168.1.17/cgi-bin/pastepile.cgi">Redirect</a></p>

</BODY>
</HTML>

paste-pile-mover.sh

A helper script called pastepile-file-mover.sh, moves the pastepile.html, the data file created by pastepile.cgi from it’s default location to a date stamped file in a location set in the script BASEDIR/dir/filename. Where BASEDIR is your choice ( I use /var/www/p/), dir is the current year and filename is YYYYMMDD.html, so that that there is a  year and date hierarchy. I allow this script to run at the start of the month via root CRON to “clean” out the Pastepile and archive the old one, much the same way that log are rotated.

0 0 1 * * /home/erick/bin/pastepile-file-mover.sh

pastepile-file-mover.sh

#!/bin/bash

# Move the pastepile.html file from it's default location to a date stamped
# file in a location BASEDIR/dir/filename
# So that it has year and date heirarchy

# For now, Monthly, Move pastepile over to year dir and datestamped HTML file
#0 0 1 * * /home/erick/bin/pastepile-file-mover.sh



BASEDIR=/var/www/p
# Testing
#BASEDIR=/tmp

dir=$(date +"%Y")
#echo $dir

#File name timestamped
filename=$(date +"%Y%m%d").html
#echo $filename

# Make the YEAR dir if it dow not exist.
if [ ! -d "$BASEDIR/$dir" ]; then
  # Control will enter here if $DIRECTORY doesn't exist.
  mkdir $BASEDIR/$dir
fi

# Do the move to the YEAR directory with the YYYYMMDD.html filename.
mv $BASEDIR/pastepile.html $BASEDIR/$dir/$filename

# This is needed make a new pastepile.html and chmod 666
# else pastepile.cgi does not work, it can't make it's own output file.
touch $BASEDIR/pastepile.html
chmod 666 $BASEDIR/pastepile.html

Last but not least pastepile.cgi

#!/usr/bin/perl -Tw

# pastepile.cgi a fork of...
# guestbook.cgi - a simple guestbook script

# This program is copyright 1999 by John Callender.

# This program is free and open software. You may use, copy, modify,
# distribute and sell this program (and any modified variants) in any way
# you wish, provided you do not restrict others from doing the same.

# pastepile.cgi - guestbook.cgi, modded to become # a pastepile program. Erick Clasen Jan 25,2018
# This new version is a dirt simple CGI program that creates a blog like
# pasting tool to be able to paste notes online or on a local server.
# allows grabbing snippets of text or code for later use on whether
# on the local machine or another on the network.

$data_file = '/var/www/p/pastepile.html';

$max_entries = 0; # how many guestbook entries to save?
                   # set to '0' (zero) for infinite entries...

use CGI;
use Fcntl;
$query = new CGI;

unless ($action = $query->param('action')) {
    $action = 'none';
}

print <<"EndOfText";
Content-type: text/html

<HTML>
<HEAD>
<TITLE>Raspberry Pi Server Paste Pile</TITLE>
</HEAD>
<BODY>
<H1>Raspberry Pi Server Paste Pile</H1> 
<P><EM>$ENV{DOCUMENT_ROOT}/p/</EM></P>



<A HREF="../status/index.html">Back to Status Index</A>&nbsp;
<A HREF="../p/2018">2018 Paste Archive</A>

<P>You can <A HREF="#form">add your own subject and entry</A> using the form at the bottom of the page. Here we is has your pastes...</P>

<HR>
EndOfText

# Input action to add a new entry. ----------------------
if ($action eq 'Add entry') {

    # process the form submission
    # and assemble the guestbook entry


    # Input the subect and the paste which is called a comment here in this code.
    $subject = $query->param('subject');

    $comment = $query->param('comment');

    # clean up and fiddle with $subject
 unless ($subject) {
        $subject = 'No Subject';
   if (length($subject) > 50) {
        $subject = 'Subject line too long >50 chars';
    }
# End Input action to add a new entry. ----------------------



    }

    # Add a time stamp, put in variable theTime. This allows the paste to be timestamped.
    
    @months = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
    @weekDays = qw(Sun Mon Tue Wed Thu Fri Sat Sun);
    @digits = qw(00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 59 59);
    ($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek) = localtime();
    $year = 1900 + $yearOffset;
    $theTime = "$digits[$hour]:$digits[$minute]:$digits[$second], $weekDays[$dayOfWeek] $months[$month] $dayOfMonth, $year";

    # untaint variable
    unless ($theTime =~ /^([^<]*)$/) {
        die "couldn't untaint name: $theTime\n";
    }
    $theTime = $1;

    

    # clean up and fiddle with $subject--------------------------------

    $subject_clean = "$subject";
    $subject_clean =~ s/, , /, /;        # remove duplicate ', '
    $subject_clean =~ s/^, //;           # remove initial ', '
    $subject_clean =~ s/, $//;           # remove final ', '
    if ($subject_clean =~ /^[,\s]+$/) {
        # nothing but commas and whitespace
        $subject_clean = 'Subject format wrong!!';
    }
    
    if (length($subject_clean) > 75) {
        $subject_clean = 'Subject too long.';
    }

    # disable HTML tags
    $subject_clean =~ s/</&lt;/g;

    # untaint variable
    unless ($subject_clean =~ /^([^<]*)$/) {
        die "couldn't untaint subject_clean: $subject_clean\n";
    }
    $subject_clean = $1;

    

    # clean up and fiddle with $comment----------------------------

    if (length($comment) > 32768) {
        $comment = '...overloaded blog buffer chars > 32768.';
    }
    unless ($comment) {
        $comment = '...nothing to speak of.';
    }

    # fix line-endings
    $comment =~ s/\r\n?/\n/g;

    # lose HTML tags
    $comment =~ s/</&lt;/g;

    # untaint variable
    unless ($comment =~ /^([^<]*)$/) {
        die "couldn't untaint comment: $comment\n";
    }
    $comment = $1;
    # end cleanup #comment-----------------------------------------

    

    # assemble finished guestbook entry -- anything after this line until EndofText will show in the post!!!!!

    # Enviroment variable for REMOTE_ADDR is grabbed and printed directly. No untainting, but probably safe to do.

    $entry = <<"EndOfText";

<P><STRONG> Subject: $subject_clean </STRONG> at <EM>$theTime </EM> from <EM>$ENV{REMOTE_ADDR} </EM> <BR>
<BLOCKQUOTE>$comment</BLOCKQUOTE></P>
<HR>
EndOfText

    # open non-destructively, read old entries, write out new
   $all_entries .= $entry;
    sysopen(ENTRIES, "$data_file", O_RDWR)
                             or die "can't open $data_file: $!";
    flock(ENTRIES, 2)        or die "can't LOCK_EX $data_file: $!";
    while(<ENTRIES>) {
        $all_entries .= $_;
    }

 if ($max_entries) {

          # lop the tail off the guestbook, if necessary

          @all_entries = split(/<HR>/i, $all_entries);
          $entry_count = @all_entries - 1;

          while ($entry_count > $max_entries) {
              pop @all_entries;
              $entry_count = @all_entries - 1;
          }

          $all_entries = join('<HR>', @all_entries);

      }


   

    # now write out to $data_file

    seek(ENTRIES, 0, 0)        or die "can't rewind $data_file: $!";
    truncate(ENTRIES, 0)       or die "can't truncate $data_file: $!";
    print ENTRIES $all_entries or die "can't print to $data_file: $!";
    close(ENTRIES)             or die "can't close $data_file: $!";

}

# display the guestbook a.k.a pastepile.html

open (IN, "$data_file") or die "Can't open $data_file for reading: $!";
flock(IN, 1)            or die "Can't get LOCK_SH on $data_file: $!";
while (<IN>) {
    print;
}
close IN                or die "Can't close $data_file: $!";

# display the form    

print <<"EndOfText";
<A NAME="form"><H2>Add a comment/entry to the paste pile (no HTML):</H2></A>

<FORM METHOD="POST" ACTION="pastepile.cgi">
<TABLE>



<TR>
<TD ALIGN="right"><STRONG>Subject: </STRONG></TD>
<TD><INPUT NAME="subject" SIZE=30></TD>
</TR>

<TR>
<TD ALIGN="right"><STRONG>Entry :</STRONG></TD>
<TD>
<TEXTAREA NAME="comment" ROWS=5 COLS=30 WRAP="virtual"></TEXTAREA>
</TD>
</TR>

<TR><TD COLSPAN=2> </TD></TR>
<TR>
<TD> </TD>
<TD><INPUT TYPE="submit" NAME="action" VALUE="Add entry"></TD>
</TR>
</TABLE>

</FORM>
</BODY>
</HTML>
EndOfText

 

Getting CGI and Perl scripts up and running on Minimal Ubuntu

I was trying, again to get this up and running. I have a piece of code notestack-example.cgi that uses Perl and the Perl Module for CGI. I had this working after fiddling with it the first time I flashed the SD card that I set up for a Pine 64 that was set up with minimal Ubuntu Xenial.

The problem is I wrote down only sketchy instructions and had to re-figure it out. After flashing another card, the first one had a slow moving corruption that would cause the machine to halt after a while, I got more clear with the process.

I have posted it here for myself, if I get stuck again and for anyone else who might need to know the process, if they get stuck. It is a rough outline. I copied what commands I issued from the history and added comments and some test code that I had on a RaspberryPi which has been running the notestack-example.cgi among other items for years so that was my baseline.

Outline getting CGI and Perl running for Apache

In this outline it is assumed that Apache2 is installed and configured.

Optional: To make it easier to get to the cgi directory from your home, create a symlink in the user home directory to be able to move into the cgi folder easier.

ln -s /usr/lib/cgi-bin cgi

Enable CGI support for Apache…
sudo a2enmod cgi

modify /etc/apache2/sites-enabled/000-default.conf

Putting in the code that enables cgi in the Apache config file for sites, this was take from the Raspberrypi and added into /etc/apache2/sites-enabled/000-default.conf
it was put right above the </VirtualHost> line.

——————————————————
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory “/usr/lib/cgi-bin”>
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

——————————————————

Test with a simple BASH CGI script

Use the following code as printenv.cgi to test basic CGI with a bash script.
in the /usr/lib/cgi-bin directory.
——————————————————
#!/bin/bash -f
echo “Content-type: text/html”
echo “”
echo “<PRE>”
env || printenv
echo “</PRE>”
exit 0
——————————————————

test with…
which needs to be 755 (+x) permissions. This file is from the RPI, was used years ago to test it out.

sudo nano printenv.cgi
sudo chmod +x printenv.cgi
  /printenv.cgi
If all is well this will also be accessible from the web…
curl http://localhost/cgi-bin/printenv.cgi

curl is installed in the minimal build.

If you are using this on your own intranet OK, if this machine is accessible from the web get rid of printenv.cgi so the whole world can’t see the output if it gets found.

 

Test Perl

Test Perl scripts, normally Perl is installed even in the minimal Ubuntu build. It’s
presence can be tested for by using ‘which perl‘.
Use the following code as perl-test.cgi
in the /usr/lib/cgi-bin directory.
——————————————————
#!/usr/bin/perl
print “Content-type: text/html\n\n”;
print “Hello CGI\n”;
EOF
——————————————————

sudo nano perl-test.cgi
 sudo chmod 755 perl-test.cgi
 ./perl-test.cgi

Does it work via Apache?
  curl http://localhost/cgi-bin/perl-test.cgi

Perl CGI Module

Get Perl scripts (Any Perl code that uses CGI.pm in it) running and web accessible, ones that require the CGI Perl Module…

Got ERROR #1,missing CGI.pm, followed by #2 complained about a missing Util.pm, see below.

Got this error when the Perl Module was missing got a similar error after installing PM, complaint about missing CGI/Util.pm
ERROR 1 and 2
Can’t locate CGI.pm in @INC (you may need to install the CGI module) (@INC contains: /etc/perl /usr/local/lib/aarch64-linux-gnu/perl/5.22.1 /usr/local/share/$
BEGIN failed–compilation aborted at ./notestack-example.cgi line 36.
Grabbed the CGI.pm and Util.pm from my Raspberry Pi by searching for them….

sudo find / -name CGI.pm

sudo find / -name Util.pm

and copying using rcp to/tmp on the board I was setting up, a Pine 64 in this case.

rcp /usr/share/perl/5.14.2/CGI.pm ubuntu@192.168.1.31:/tmp
rcp /usr/share/perl/5.14.2/CGI/Util.pm ubuntu@192.168.1.31:/tmp

The code I was trying to run, the goal of all this work was a script named notestack-example.cgi.
sudo cp /tmp/CGI.pm /etc/perl
./notestack-example.cgi      <— Got ERROR #1
sudo cp /tmp/Util.pm /etc/perl/
./notestack-example.cgi      <— Got ERROR #2
sudo mkdir /etc/perl/CGI
sudo mv /etc/perl/Util.pm /etc/perl/CGI

Final error ERROR 3

Can’t use ‘defined(@array)’ (Maybe you should just omit the defined()?) at /etc/perl/CGI.pm line 528.
Compilation failed in require at ./notestack-example.cgi line 36.
BEGIN failed–compilation aborted at ./notestack-example.cgi line 36.

This one requires removing defined as this is old and not compatible with the current version of Perl.
Just removed the defined on line 528…

ubuntu@pine64:~$ diff /etc/perl/CGI.pm /tmp/CGI.pm
528c528
< if (@QUERY_PARAM && !$initializer) {

> if (defined(@QUERY_PARAM) && !defined($initializer)) {

Learned about this trick about removing the ‘defined‘ from…
https://github.com/shenlab-sinai/diffreps/issues/6
https://rt.cpan.org/Public/Bug/Display.html?id=79917

PHPList config.php Tweaks

PHPList Config File

PHPList works great for large email lists but, I found out that the configuration file can be tweaked to make certain aspects of it work even better.

The file to modify is config.php located under the PHPList/config folder which should be underneath the public_html folder on most installations.

 Batch Processing Tweak

I have found that I have to frequently hit process queue on PHPList to get the entire email list to process. Recently I found that changing the following settings has made a big improvement in it’s performance. My service provider limits emails to 500 per hour so I set the MAILQUEUE_BATCH_SIZE at 480 and the batch period at 3600. Now it will send out the messages as fast as possible up to the limit of 480. It seems to perform better if it can “get the messages out” fairly quickly. It will send out a bunch with a 2 second spacing that I have set ( see below ) and it seems to get them out most of the time before what ever timing was running out that was making me have hit process queue again.

 

## if you send the queue using commandline, you can set it to something that complies with the
## limits of your ISP, eg 300 messages an hour would be 
define("MAILQUEUE_BATCH_SIZE",480);
define("MAILQUEUE_BATCH_PERIOD",3600);

Make sure the MAILQUEUE_BATCH_SIZE and MAILQUEUE_BATCH_PERIOD is commented out, it might conflict with what is done above….

## if you send the queue using your browser, you may want to consider settings like this
## which will send 10 messages and then reload the browser to send the next 10. However, this
## will not restrict the sending to any limits, so there's a good chance you will
## go over the limits of your ISP
#define("MAILQUEUE_BATCH_SIZE",10);
#define("MAILQUEUE_BATCH_PERIOD",1);

So I don’t overload the server by trying to send out messages really fast I have set a slight delay…

# to avoid overloading the server that sends your email, you can add a little delay
# between messages that will spread the load of sending
# you will need to find a good value for your own server
# value is in seconds, and you can use fractions, eg "0.5" is half a second
# (or you can play with the autothrottle below)
define('MAILQUEUE_THROTTLE',2);

Queue Processing Reporting

It is also nice to see the results and know that the message actually went out. So I have it set to report this result…

# after every run of the queue to send out messages, phpList will send a summary to the
 # admin address. If you want to stop this, set this to false or 0
 define('SEND_QUEUE_PROCESSING_REPORT',true);

 

Send Only One Test Email

If you send a lot of test emails to yourself and it is a pain to get a text and HTML every time you can restrict it to just sending one. The one it will send is based on the way the user preferences are set up.

# test emails
 # if you send a test email, phplist will by default send you two emails, one in HTML format
 # and the other in Text format. If you set this to 1, you can override this behaviour
 # and only have a test email sent to you that matches the user record of the user that the
 # test emails are sent to
 define('SEND_ONE_TESTMAIL',1);

 Click Tracking

Click tracking is a method of tracking where email recipients go based on the links in the body of the email. You can track a decent amount of information on who goes to what links and even the timing of it. I have found this feature very useful as it can give an idea of how well your email is penetrating the market. To use this feature you have to use the [CLICKTRACK] directive in the message body or in a template.

# Click tracking
# If you set this to 1, all links in your emails will be converted to links that
# go via phplist. This will make sure that clicks are tracked. This is experimental and
# all your findings when using this feature should be reported to mantis
# for now it's off by default until we think it works correctly
define('CLICKTRACK',1);

# Click track, list detail
# if you enable this, you will get some extra statistics about unique users who have clicked the
# links in your messages, and the breakdown between clicks from text or html messages.
# However, this will slow down the process to view the statistics, so it is
# recommended to leave it off, but if you're very curious, you can enable it
define('CLICKTRACK_SHOWDETAIL',1);

 

Attachments

It is worth turning on the attachments feature in PHPList to be able to add attachments to the email. Turning it on creates a new tab in the PHPList menu while you are creating a message that allows you to add an attachment.

# attachments is a new feature and is currently still experimental
# set this to 1 if you want to try it
# caution, message may become very large. it is generally more
# acceptable to send a URL for download to users
# if you try it, it will be appreciated to give feedback to the
# users mailinglist, so we can learn whether it is working ok
# using attachments requires PHP 4.1.0 and up
define("ALLOW_ATTACHMENTS",1);

# if you use the above, how many would you want to add per message (max)
# You can leave this 1, even if you want to attach more files, because
# you will be able to add them sequentially
define("NUMATTACHMENTS",1);

# when using attachments you can upload them to the server
# if you want to use attachments from the local filesystem (server) set this to 1
# filesystem attachments are attached at real send time of the message, not at
# the time of creating the message
define("FILESYSTEM_ATTACHMENTS",1);

Bounce Processing

I have set the purge and purge unprocessed to 1 that way the mailbox doesn’t fill up. I want it to empty when the bounces are processed every month. I set the unsubscribe threshold to 15 instead of the normal 5. This way it really gives a lot of chances for the person to straighten out their email before it stops sending to them. It is no big deal to let it send mail that will bounce back for a while, in my opinion. But eventually I want the mailing list cleaned of deadwood, even if it takes a year it is OK.

# set this to 0 if you want to keep your messages in the mailbox. this is potentially
 # a problem, because bounces will be counted multiple times, so only do this if you are
 # testing things.
 $bounce_mailbox_purge = 1;
# set this to 0 if you want to keep unprocessed messages in the mailbox. Unprocessed
 # messages are messages that could not be matched with a user in the system
 # messages are still downloaded into PHPlist, so it is safe to delete them from
 # the mailbox and view them in PHPlist
 $bounce_mailbox_purge_unprocessed = 1;
# how many bounces in a row need to have occurred for a user to be marked unconfirmed
 $bounce_unsubscribe_threshold = 15;

Wishlist

I would like to find a way to increase the timeout for PHPList when editing a message. The timeout that I am referring to is the one that occurs if you don’t save the draft message periodically and one hour goes by. Then you wind up losing your work as the PHPList makes you resign in. So if you forget to save or get interrupted and come back later, a lot of work can be lost accidentally. Even if it had a feature such as WordPress where it periodically saved a draft, that would be a big help.

Automating PHPList Process Queue

Frequently when using the non command line version of PHPList which I have loaded on my service providers server, I have to hit Process Queue multiple times for a message. I have read a lot of posts online complaining this issue. I have done some troubleshooting of it myself by turning on the verbose feature in PHPList and I am still at a loss as to what it gets stuck on and then stops processing the queue.

PHPList works great, I love it for all that it does and does well. My one complaint is the Process Queue issue and from reading a bunch of posts online I have successfully patched together a solution.

This code can be executed on the server where PHPList lives if they will let you run a wget command from a CRON entry, some won’t. In my case I opted to simply run the command from a Raspberry Pi server that I have that runs 24/7. It is simple a looping script that does the equivalent of hitting the “Process Queue” button on PHPList at regularly timed intervals and detected when the queue has been processed. I have seen several examples of using wget to be fired off every 5 minutes to do this, but that means it gets run every 5 minutes, all day, every day. Why not run something that can be scheduled for a time and then runs to completion which this script does.
This script also produces a log file that can be copied to a location viewable via the server so you can double check to see if all ran OK. I see this mostly as a debug feature and it could potentially be turned off along with some of the “echos” of text. Except for the one in the else-if, it is an error to have an empty else-if in a bash script.

PHPList Automatic Process Queue Script

#!/bin/bash
 # Controller Script for PHPlist Process Queue
 # Erick Clasen blog.oils-of-life.com 05/16/2015
# User defined variables
 # No trailing / on DIR!
 DIR=/home/erick/php-proc
 LOG=php-pq.log
 # Time to sleep between loop iterations.
 TIME=3m
 # Max number of loops, so the script breaks out if for some reason it fails to complete.
 LOOPS=100
 # Publish a copy of the log to the following directory
 PUBDIR=/home/erick/public_html
# wget having trouble with some options so simply switch to the dir we want it to work in.
 cd $DIR
x=1
 echo Start------------- >> $LOG
# While loop, breaks out if done or at LOOPS as a guard against #infinite loop.
 while [ $x -le $LOOPS ]; do
# Timestamp in DDMMYYYY_HHMM-SS (UTC)
 timestamp=$(date -u +"%d%m%Y_%H%M-%S")
echo $x $timestamp
 echo $x $timestamp >> $LOG
x=$(( $x + 1 ))
# Use wget to run process queue with login and password provided
 wget 'your_url_here.com/phpList/admin/?page=processqueue&login=phplist&password=your_password_here'
# Test to see if PHPlist process queue has completed.
 if grep -q "Finished, Nothing to do" $DIR/index.*
 then
 # Exit from Loop with a break, mark log,remove index files that wget fetched, break out.
 echo All Done! Breaking Out
 echo All Done!--------- >> $LOG
 echo ------------------ >> $LOG
 # Publish Log, Optional
 cp $LOG $PUBDIR/$LOG
 rm $DIR/index.*
 break
 else
 # Nothing Happens
 echo Keep Running...
 fi
# Sleep for TIME
 echo ------- Sleeping Process --------
sleep $TIME;
done;

PHPList Automated Process Bounces Script

Process Bounces can be run from a script as well…

#!/bin/bash
 # Controller Script for PHPlist Process Bounces
 # Erick Clasen blog.oils-of-life.com 05/16/2015
# User defined variables
 # No trailing / on DIR!
 DIR=/home/erick/php-proc
 LOG=php-pq.log
# Publish a copy of the log to the following directory
 PUBDIR=/home/erick/public_html
# wget having trouble with some options so simply switch to the dir we want it $
 cd $DIR
# Use wget to run process queue with login and password provided
 wget 'your_url_here.com/phpList/admin/?page=processbounces&login=phplist&password=your_password_here'
 echo ---------------------------- >> $LOG
 echo BOUNCES HAVE BEEN PROCESSED! >> $LOG
 echo ---------------------------- >> $LOG
 # Publish Log, Optional
 cp $LOG $PUBDIR/$LOG
 rm $DIR/index.*

CRON entry to run these scripts

These entries run Process Queue every Monday, Wednesday and Friday (1,3,5) at 4:15AM and Process Bounces at 12:15AM on the first day of the month. Nothing magic about those settings, you can set them any way you wish. I figure starting the process in the wee hours of the morning will allow people to receive the email by the time that they wake and first check their mail. I set the time to 15 after so it won’t run at the same time as some processes that I have set to run on the hour. Sending mail on weekdays keeps people from having to deal with more email than they might like on the weekends.

15 04 * * 1,3,5 /home/erick/php-proc/php-pq.sh
 15 00 1 * * /home/erick/php-proc/php-pb.sh

PHPList Automatic Process Queue Override

If you must make PHPList process the queue now and don’t want to wait for the CRON task. Two options are available, do it the via the normal way from the PHPList admin website or you can use something as simple as a CGI script to force the php-pq.sh script to execute. Below is an example of a dirt simple CGI script. It prints out the tail of the php-pq.log and runs the php-pq.sh script. Php-pq.log is the log of the previous automatic processing event and then goes ahead and fires off the Process Queue Override by running the script. Basically the act of loading the page makes it run. I keep my cgi-bin directory locked down so no one outside of a few IP addresses can get to the files there. But it would be fairly harmless if someone did, the worst that happens is that the queue gets processed.

CGI Code for Automatic Process Queue Override

#!/bin/bash
 echo "Content-type: text/html"
 echo ""
 echo "PHPList Automatic Process Queue Override"
 echo ""
 echo "
PHPList Automatic Process Queue Override from host $(hostname -s &)
"
 echo ""
 echo "
 $(tail /home/erick/php-proc/php-pq.log &)
"
 echo ""
 echo "
Information generated on $(date &)
"
 echo ""
 echo "Processing..."
echo "
 $(/home/erick/php-proc/php-pq.sh &)
"
echo "
"

 

PHP Config File Tweaks

I am working on taking some notes on some changes I have made to the PHPList config.php file and will be posting on this in August 2015.