Tag Archives: CGI

Penguin

guestbook.cgi

The best way to understand a thing is to try to comprehend it.

I ran John Callendar’s guestbook.cgi on one of my servers. It wasn’t a very popular location on the web and the guestbook did not get abused fortunately! It would be very easy for comment spammers, trollers and the like to have a field day with it.
One thing I did to keep an eye on it and see if it got posted to was a script in /etc/cron.weekly to check for updates of the guestbook. It was located on a Raspberry Pi running Raspberian.
The script requires that ssmtp or some mail program is installed.

guestbook-check.sh

#!/bin/bash

# Checks to see if the Guestbook has been written to in the past week.
 # Sends out notification if it has been written to.
 find /usr/lib/cgi-bin/guestbookrev.txt -mtime -7 -exec mail -s "Guestbook Updated" myemail@myisp.net \;

The trick is in the find command where the option -mtime -7 means check to see if the file have been has a modification time of less than 7 days, if so, then execute via the -exec option, whatever comes next on the command line until the \;

Subject line

The mail command mails only a subject line of “Guestbook Updated”. It would be possible to have something in the body and even “cat” the guestbook in to send it in the email.

Auto de Spam

If the guestbook got abused by a spammer or something nasty it might be possible to run a script that would periodically do a cleanup on the file via a search and replace. Using a list of blacklisted words to search on and then replace them with a null character or space.

Reverse Order in Guestbook

It is possible to flip the posts on the guestbook as well if you want them ordered in the opposite order, Last In On Top versus the default of First In On Top. This is done by using…

 pop @all_entries instead of shift @all_entries

…in the code.

More on CGI and Perl

If you are new to CGI and/or Perl scripts be sure to check out John Callender’s tutorial that covers the workings of guestbook.cgi.
guestbook.cgi

Penguin

Pastepile

Future development depends on the determination of future generations to contribute towards building their own society.

Pastepile is a fork of guestbook.cgi – a simple guestbook script, written by John Callender in 1999. It is the last part of his Beginner’s Guide to CGI Scripting with Perl, Running a Guestbook.
This new version, a.k.a Pastepile, is a dirt simple Perl CGI program that creates a blog like pasting tool to be able to paste notes online or on a local server. Allows grabbing snippets of text or code for later use on whether on the local machine or another on the network. I made it sometime after studying John Callender’s tutorial when I was digging into learning about CGI and Perl several years ago.

History

Pastepile originated as a way to paste snippets of code and write short notes on configuration changes that I make on servers, including a Raspberry Pi that I run. The only requirement is the installation of a web server such as Apache. This little Pastepile tool made it easy to keep track of information related to the servers in one place  and search-able via a browser. After using it in the is mode for a while, I cloned it and started to use it as a general place to paste info and write short notes. This makes it easy to save something while on one machine on the network and be able to retrieve it later on that machine or another.

Example of What it Looks Like


Subject: Running a form-to-email gateway at 19:04:44, Sat Feb 3, 2018 from 192.168.1.7

http://www.resoo.org/docs/perl/begperl/form_to_email.html


Subject: Slackware Linux Essentials at 12:10:03, Thu Feb 1, 2018 from 192.168.1.179

www.slackbook.org/html/index.html


Additions:Remote Address,Time,Reversed Order

Besides stripping down guestbook.cgi to remove code specifc to it’s guestbook function and cleaning up the code to reflect it’s new use, three functional code changes were made. One is to add a timestamp to keep track of when the entry was created. Another change made was showing the IP of the machine the post originates from by reading the environment variable $REMOTE_ADDR to keep track of where the post originates from. Many machines on my network have static IP’s so this is handy for me at least. The final change was to reverse the order of the listing. In guestbook.cgi a First In On Top ordering is the layout of the guestbook posts. For Pastepile I reversed it so the most recent and not the oldest post is on top of the list, Last In On Top ordering. This made sense as the most recently written notes are more likely to be important and I want those to be close to the top of the pile and let the file get long with the aging information at the bottom.

/p/

For ease of access via a shorter link a simple index.html redirector page was place in /var/www/p/, following a 4chan-esqe style of folder naming and a nice short link to get to the pastepile.cgi script, which lives in the /usr/lib/cgi-bin/ directory. The /p/ directory also holds the pastepile.html file that is created by pastepile.cgi as it’s data page.

 Redirector Example for /var/www/p/index.html

<HTML>
<HEAD>
<meta http-equiv="refresh" content="0; url='http://192.168.1.17/cgi-bin/pastepile.cgi'"/>
</HEAD>
<BODY>
<p><a href="http://192.168.1.17/cgi-bin/pastepile.cgi">Redirect</a></p>

</BODY>
</HTML>

paste-pile-mover.sh

A helper script called pastepile-file-mover.sh, moves the pastepile.html, the data file created by pastepile.cgi from it’s default location to a date stamped file in a location set in the script BASEDIR/dir/filename. Where BASEDIR is your choice ( I use /var/www/p/), dir is the current year and filename is YYYYMMDD.html, so that that there is a  year and date hierarchy. I allow this script to run at the start of the month via root CRON to “clean” out the Pastepile and archive the old one, much the same way that log are rotated.

0 0 1 * * /home/erick/bin/pastepile-file-mover.sh

pastepile-file-mover.sh

#!/bin/bash

# Move the pastepile.html file from it's default location to a date stamped
# file in a location BASEDIR/dir/filename
# So that it has year and date heirarchy

# For now, Monthly, Move pastepile over to year dir and datestamped HTML file
#0 0 1 * * /home/erick/bin/pastepile-file-mover.sh



BASEDIR=/var/www/p
# Testing
#BASEDIR=/tmp

dir=$(date +"%Y")
#echo $dir

#File name timestamped
filename=$(date +"%Y%m%d").html
#echo $filename

# Make the YEAR dir if it dow not exist.
if [ ! -d "$BASEDIR/$dir" ]; then
  # Control will enter here if $DIRECTORY doesn't exist.
  mkdir $BASEDIR/$dir
fi

# Do the move to the YEAR directory with the YYYYMMDD.html filename.
mv $BASEDIR/pastepile.html $BASEDIR/$dir/$filename

# This is needed make a new pastepile.html and chmod 666
# else pastepile.cgi does not work, it can't make it's own output file.
touch $BASEDIR/pastepile.html
chmod 666 $BASEDIR/pastepile.html

Last but not least pastepile.cgi

#!/usr/bin/perl -Tw

# pastepile.cgi a fork of...
# guestbook.cgi - a simple guestbook script

# This program is copyright 1999 by John Callender.

# This program is free and open software. You may use, copy, modify,
# distribute and sell this program (and any modified variants) in any way
# you wish, provided you do not restrict others from doing the same.

# pastepile.cgi - guestbook.cgi, modded to become # a pastepile program. Erick Clasen Jan 25,2018
# This new version is a dirt simple CGI program that creates a blog like
# pasting tool to be able to paste notes online or on a local server.
# allows grabbing snippets of text or code for later use on whether
# on the local machine or another on the network.

$data_file = '/var/www/p/pastepile.html';

$max_entries = 0; # how many guestbook entries to save?
                   # set to '0' (zero) for infinite entries...

use CGI;
use Fcntl;
$query = new CGI;

unless ($action = $query->param('action')) {
    $action = 'none';
}

print <<"EndOfText";
Content-type: text/html

<HTML>
<HEAD>
<TITLE>Raspberry Pi Server Paste Pile</TITLE>
</HEAD>
<BODY>
<H1>Raspberry Pi Server Paste Pile</H1> 
<P><EM>$ENV{DOCUMENT_ROOT}/p/</EM></P>



<A HREF="../status/index.html">Back to Status Index</A>&nbsp;
<A HREF="../p/2018">2018 Paste Archive</A>

<P>You can <A HREF="#form">add your own subject and entry</A> using the form at the bottom of the page. Here we is has your pastes...</P>

<HR>
EndOfText

# Input action to add a new entry. ----------------------
if ($action eq 'Add entry') {

    # process the form submission
    # and assemble the guestbook entry


    # Input the subect and the paste which is called a comment here in this code.
    $subject = $query->param('subject');

    $comment = $query->param('comment');

    # clean up and fiddle with $subject
 unless ($subject) {
        $subject = 'No Subject';
   if (length($subject) > 50) {
        $subject = 'Subject line too long >50 chars';
    }
# End Input action to add a new entry. ----------------------



    }

    # Add a time stamp, put in variable theTime. This allows the paste to be timestamped.
    
    @months = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
    @weekDays = qw(Sun Mon Tue Wed Thu Fri Sat Sun);
    @digits = qw(00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 59 59);
    ($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek) = localtime();
    $year = 1900 + $yearOffset;
    $theTime = "$digits[$hour]:$digits[$minute]:$digits[$second], $weekDays[$dayOfWeek] $months[$month] $dayOfMonth, $year";

    # untaint variable
    unless ($theTime =~ /^([^<]*)$/) {
        die "couldn't untaint name: $theTime\n";
    }
    $theTime = $1;

    

    # clean up and fiddle with $subject--------------------------------

    $subject_clean = "$subject";
    $subject_clean =~ s/, , /, /;        # remove duplicate ', '
    $subject_clean =~ s/^, //;           # remove initial ', '
    $subject_clean =~ s/, $//;           # remove final ', '
    if ($subject_clean =~ /^[,\s]+$/) {
        # nothing but commas and whitespace
        $subject_clean = 'Subject format wrong!!';
    }
    
    if (length($subject_clean) > 75) {
        $subject_clean = 'Subject too long.';
    }

    # disable HTML tags
    $subject_clean =~ s/</&lt;/g;

    # untaint variable
    unless ($subject_clean =~ /^([^<]*)$/) {
        die "couldn't untaint subject_clean: $subject_clean\n";
    }
    $subject_clean = $1;

    

    # clean up and fiddle with $comment----------------------------

    if (length($comment) > 32768) {
        $comment = '...overloaded blog buffer chars > 32768.';
    }
    unless ($comment) {
        $comment = '...nothing to speak of.';
    }

    # fix line-endings
    $comment =~ s/\r\n?/\n/g;

    # lose HTML tags
    $comment =~ s/</&lt;/g;

    # untaint variable
    unless ($comment =~ /^([^<]*)$/) {
        die "couldn't untaint comment: $comment\n";
    }
    $comment = $1;
    # end cleanup #comment-----------------------------------------

    

    # assemble finished guestbook entry -- anything after this line until EndofText will show in the post!!!!!

    # Enviroment variable for REMOTE_ADDR is grabbed and printed directly. No untainting, but probably safe to do.

    $entry = <<"EndOfText";

<P><STRONG> Subject: $subject_clean </STRONG> at <EM>$theTime </EM> from <EM>$ENV{REMOTE_ADDR} </EM> <BR>
<BLOCKQUOTE>$comment</BLOCKQUOTE></P>
<HR>
EndOfText

    # open non-destructively, read old entries, write out new
   $all_entries .= $entry;
    sysopen(ENTRIES, "$data_file", O_RDWR)
                             or die "can't open $data_file: $!";
    flock(ENTRIES, 2)        or die "can't LOCK_EX $data_file: $!";
    while(<ENTRIES>) {
        $all_entries .= $_;
    }

 if ($max_entries) {

          # lop the tail off the guestbook, if necessary

          @all_entries = split(/<HR>/i, $all_entries);
          $entry_count = @all_entries - 1;

          while ($entry_count > $max_entries) {
              pop @all_entries;
              $entry_count = @all_entries - 1;
          }

          $all_entries = join('<HR>', @all_entries);

      }


   

    # now write out to $data_file

    seek(ENTRIES, 0, 0)        or die "can't rewind $data_file: $!";
    truncate(ENTRIES, 0)       or die "can't truncate $data_file: $!";
    print ENTRIES $all_entries or die "can't print to $data_file: $!";
    close(ENTRIES)             or die "can't close $data_file: $!";

}

# display the guestbook a.k.a pastepile.html

open (IN, "$data_file") or die "Can't open $data_file for reading: $!";
flock(IN, 1)            or die "Can't get LOCK_SH on $data_file: $!";
while (<IN>) {
    print;
}
close IN                or die "Can't close $data_file: $!";

# display the form    

print <<"EndOfText";
<A NAME="form"><H2>Add a comment/entry to the paste pile (no HTML):</H2></A>

<FORM METHOD="POST" ACTION="pastepile.cgi">
<TABLE>



<TR>
<TD ALIGN="right"><STRONG>Subject: </STRONG></TD>
<TD><INPUT NAME="subject" SIZE=30></TD>
</TR>

<TR>
<TD ALIGN="right"><STRONG>Entry :</STRONG></TD>
<TD>
<TEXTAREA NAME="comment" ROWS=5 COLS=30 WRAP="virtual"></TEXTAREA>
</TD>
</TR>

<TR><TD COLSPAN=2> </TD></TR>
<TR>
<TD> </TD>
<TD><INPUT TYPE="submit" NAME="action" VALUE="Add entry"></TD>
</TR>
</TABLE>

</FORM>
</BODY>
</HTML>
EndOfText

 

Getting CGI and Perl scripts up and running on Minimal Ubuntu

I was trying, again to get this up and running. I have a piece of code notestack-example.cgi that uses Perl and the Perl Module for CGI. I had this working after fiddling with it the first time I flashed the SD card that I set up for a Pine 64 that was set up with minimal Ubuntu Xenial.

The problem is I wrote down only sketchy instructions and had to re-figure it out. After flashing another card, the first one had a slow moving corruption that would cause the machine to halt after a while, I got more clear with the process.

I have posted it here for myself, if I get stuck again and for anyone else who might need to know the process, if they get stuck. It is a rough outline. I copied what commands I issued from the history and added comments and some test code that I had on a RaspberryPi which has been running the notestack-example.cgi among other items for years so that was my baseline.

Outline getting CGI and Perl running for Apache

In this outline it is assumed that Apache2 is installed and configured.

Optional: To make it easier to get to the cgi directory from your home, create a symlink in the user home directory to be able to move into the cgi folder easier.

ln -s /usr/lib/cgi-bin cgi

Enable CGI support for Apache…
sudo a2enmod cgi

modify /etc/apache2/sites-enabled/000-default.conf

Putting in the code that enables cgi in the Apache config file for sites, this was take from the Raspberrypi and added into /etc/apache2/sites-enabled/000-default.conf
it was put right above the </VirtualHost> line.

——————————————————
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory “/usr/lib/cgi-bin”>
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

——————————————————

Test with a simple BASH CGI script

Use the following code as printenv.cgi to test basic CGI with a bash script.
in the /usr/lib/cgi-bin directory.
——————————————————
#!/bin/bash -f
echo “Content-type: text/html”
echo “”
echo “<PRE>”
env || printenv
echo “</PRE>”
exit 0
——————————————————

test with…
which needs to be 755 (+x) permissions. This file is from the RPI, was used years ago to test it out.

sudo nano printenv.cgi
sudo chmod +x printenv.cgi
  /printenv.cgi
If all is well this will also be accessible from the web…
curl http://localhost/cgi-bin/printenv.cgi

curl is installed in the minimal build.

If you are using this on your own intranet OK, if this machine is accessible from the web get rid of printenv.cgi so the whole world can’t see the output if it gets found.

 

Test Perl

Test Perl scripts, normally Perl is installed even in the minimal Ubuntu build. It’s
presence can be tested for by using ‘which perl‘.
Use the following code as perl-test.cgi
in the /usr/lib/cgi-bin directory.
——————————————————
#!/usr/bin/perl
print “Content-type: text/html\n\n”;
print “Hello CGI\n”;
EOF
——————————————————

sudo nano perl-test.cgi
 sudo chmod 755 perl-test.cgi
 ./perl-test.cgi

Does it work via Apache?
  curl http://localhost/cgi-bin/perl-test.cgi

Perl CGI Module

Get Perl scripts (Any Perl code that uses CGI.pm in it) running and web accessible, ones that require the CGI Perl Module…

Got ERROR #1,missing CGI.pm, followed by #2 complained about a missing Util.pm, see below.

Got this error when the Perl Module was missing got a similar error after installing PM, complaint about missing CGI/Util.pm
ERROR 1 and 2
Can’t locate CGI.pm in @INC (you may need to install the CGI module) (@INC contains: /etc/perl /usr/local/lib/aarch64-linux-gnu/perl/5.22.1 /usr/local/share/$
BEGIN failed–compilation aborted at ./notestack-example.cgi line 36.
Grabbed the CGI.pm and Util.pm from my Raspberry Pi by searching for them….

sudo find / -name CGI.pm

sudo find / -name Util.pm

and copying using rcp to/tmp on the board I was setting up, a Pine 64 in this case.

rcp /usr/share/perl/5.14.2/CGI.pm ubuntu@192.168.1.31:/tmp
rcp /usr/share/perl/5.14.2/CGI/Util.pm ubuntu@192.168.1.31:/tmp

The code I was trying to run, the goal of all this work was a script named notestack-example.cgi.
sudo cp /tmp/CGI.pm /etc/perl
./notestack-example.cgi      <— Got ERROR #1
sudo cp /tmp/Util.pm /etc/perl/
./notestack-example.cgi      <— Got ERROR #2
sudo mkdir /etc/perl/CGI
sudo mv /etc/perl/Util.pm /etc/perl/CGI

Final error ERROR 3

Can’t use ‘defined(@array)’ (Maybe you should just omit the defined()?) at /etc/perl/CGI.pm line 528.
Compilation failed in require at ./notestack-example.cgi line 36.
BEGIN failed–compilation aborted at ./notestack-example.cgi line 36.

This one requires removing defined as this is old and not compatible with the current version of Perl.
Just removed the defined on line 528…

ubuntu@pine64:~$ diff /etc/perl/CGI.pm /tmp/CGI.pm
528c528
< if (@QUERY_PARAM && !$initializer) {

> if (defined(@QUERY_PARAM) && !defined($initializer)) {

Learned about this trick about removing the ‘defined‘ from…
https://github.com/shenlab-sinai/diffreps/issues/6
https://rt.cpan.org/Public/Bug/Display.html?id=79917

Automating PHPList Process Queue

Frequently when using the non command line version of PHPList which I have loaded on my service providers server, I have to hit Process Queue multiple times for a message. I have read a lot of posts online complaining this issue. I have done some troubleshooting of it myself by turning on the verbose feature in PHPList and I am still at a loss as to what it gets stuck on and then stops processing the queue.

PHPList works great, I love it for all that it does and does well. My one complaint is the Process Queue issue and from reading a bunch of posts online I have successfully patched together a solution.

This code can be executed on the server where PHPList lives if they will let you run a wget command from a CRON entry, some won’t. In my case I opted to simply run the command from a Raspberry Pi server that I have that runs 24/7. It is simple a looping script that does the equivalent of hitting the “Process Queue” button on PHPList at regularly timed intervals and detected when the queue has been processed. I have seen several examples of using wget to be fired off every 5 minutes to do this, but that means it gets run every 5 minutes, all day, every day. Why not run something that can be scheduled for a time and then runs to completion which this script does.
This script also produces a log file that can be copied to a location viewable via the server so you can double check to see if all ran OK. I see this mostly as a debug feature and it could potentially be turned off along with some of the “echos” of text. Except for the one in the else-if, it is an error to have an empty else-if in a bash script.

PHPList Automatic Process Queue Script

#!/bin/bash
 # Controller Script for PHPlist Process Queue
 # Erick Clasen blog.oils-of-life.com 05/16/2015
# User defined variables
 # No trailing / on DIR!
 DIR=/home/erick/php-proc
 LOG=php-pq.log
 # Time to sleep between loop iterations.
 TIME=3m
 # Max number of loops, so the script breaks out if for some reason it fails to complete.
 LOOPS=100
 # Publish a copy of the log to the following directory
 PUBDIR=/home/erick/public_html
# wget having trouble with some options so simply switch to the dir we want it to work in.
 cd $DIR
x=1
 echo Start------------- >> $LOG
# While loop, breaks out if done or at LOOPS as a guard against #infinite loop.
 while [ $x -le $LOOPS ]; do
# Timestamp in DDMMYYYY_HHMM-SS (UTC)
 timestamp=$(date -u +"%d%m%Y_%H%M-%S")
echo $x $timestamp
 echo $x $timestamp >> $LOG
x=$(( $x + 1 ))
# Use wget to run process queue with login and password provided
 wget 'your_url_here.com/phpList/admin/?page=processqueue&login=phplist&password=your_password_here'
# Test to see if PHPlist process queue has completed.
 if grep -q "Finished, Nothing to do" $DIR/index.*
 then
 # Exit from Loop with a break, mark log,remove index files that wget fetched, break out.
 echo All Done! Breaking Out
 echo All Done!--------- >> $LOG
 echo ------------------ >> $LOG
 # Publish Log, Optional
 cp $LOG $PUBDIR/$LOG
 rm $DIR/index.*
 break
 else
 # Nothing Happens
 echo Keep Running...
 fi
# Sleep for TIME
 echo ------- Sleeping Process --------
sleep $TIME;
done;

PHPList Automated Process Bounces Script

Process Bounces can be run from a script as well…

#!/bin/bash
 # Controller Script for PHPlist Process Bounces
 # Erick Clasen blog.oils-of-life.com 05/16/2015
# User defined variables
 # No trailing / on DIR!
 DIR=/home/erick/php-proc
 LOG=php-pq.log
# Publish a copy of the log to the following directory
 PUBDIR=/home/erick/public_html
# wget having trouble with some options so simply switch to the dir we want it $
 cd $DIR
# Use wget to run process queue with login and password provided
 wget 'your_url_here.com/phpList/admin/?page=processbounces&login=phplist&password=your_password_here'
 echo ---------------------------- >> $LOG
 echo BOUNCES HAVE BEEN PROCESSED! >> $LOG
 echo ---------------------------- >> $LOG
 # Publish Log, Optional
 cp $LOG $PUBDIR/$LOG
 rm $DIR/index.*

CRON entry to run these scripts

These entries run Process Queue every Monday, Wednesday and Friday (1,3,5) at 4:15AM and Process Bounces at 12:15AM on the first day of the month. Nothing magic about those settings, you can set them any way you wish. I figure starting the process in the wee hours of the morning will allow people to receive the email by the time that they wake and first check their mail. I set the time to 15 after so it won’t run at the same time as some processes that I have set to run on the hour. Sending mail on weekdays keeps people from having to deal with more email than they might like on the weekends.

15 04 * * 1,3,5 /home/erick/php-proc/php-pq.sh
 15 00 1 * * /home/erick/php-proc/php-pb.sh

PHPList Automatic Process Queue Override

If you must make PHPList process the queue now and don’t want to wait for the CRON task. Two options are available, do it the via the normal way from the PHPList admin website or you can use something as simple as a CGI script to force the php-pq.sh script to execute. Below is an example of a dirt simple CGI script. It prints out the tail of the php-pq.log and runs the php-pq.sh script. Php-pq.log is the log of the previous automatic processing event and then goes ahead and fires off the Process Queue Override by running the script. Basically the act of loading the page makes it run. I keep my cgi-bin directory locked down so no one outside of a few IP addresses can get to the files there. But it would be fairly harmless if someone did, the worst that happens is that the queue gets processed.

CGI Code for Automatic Process Queue Override

#!/bin/bash
 echo "Content-type: text/html"
 echo ""
 echo "PHPList Automatic Process Queue Override"
 echo ""
 echo "
PHPList Automatic Process Queue Override from host $(hostname -s &)
"
 echo ""
 echo "
 $(tail /home/erick/php-proc/php-pq.log &)
"
 echo ""
 echo "
Information generated on $(date &)
"
 echo ""
 echo "Processing..."
echo "
 $(/home/erick/php-proc/php-pq.sh &)
"
echo "
"

 

PHP Config File Tweaks

I am working on taking some notes on some changes I have made to the PHPList config.php file and will be posting on this in August 2015.