Category Archives: Raspberry Pi

Simple Linux Performance Benchmarking

Linux, Raspberry Pi, , , , , , , , , , , ,

Recently I did some very simple benchmarking of the CPU and disk drives on a few of my Linux PC’s and a Raspberry Pi. This was a quick test to see how all of the machines compare.

CPU: I went for some very simple tests of the CPU performance under a load, calculating Pi. The code is not the best at calculating pi, it is just there to exercise the processor(s) and provide some standard piece of executable code that can be tried on multiple machines. When the pi code runs it will use the processor at close to 100% when viewed using top.

13502 erick     20   0  2784 1964 1112 R  98.8  0.4   0:36.13 arm_pi

There are programs available that are better at calculating a lot of digits of pi, fast. An example is Hyper Pi for Windows that will certainly calculate Pi to many more digits in a given period of time than the pi shown in this post program can.

DISKS: To test disk speeds. I used a simple writing and reading out of the bitbucket ( /dev/null) to disk and noting the speeds displayed.

CPU Benchmarking

CPU benchmarking was done by running an executable that calculates Pi to an arbitrary number of digits. Below is the link to the C file which is easily compiled for the target machine. I downloaded this from the net, from this page… http://www.overclockers.com/forums/archive/index.php/t-402437.html

pi.c

For a Linux machine or the Raspberry Pi, downloading the file and executing…

gcc -o pi pi.c

or

gcc -o pi pi.c -lm

As the original post states. The -l means link a library. The m means the math library. I don’t think it is necessary???

…will create the executable pi.

To run in the directory it was create just call pi with an argument of the number of digits to calculate, for example Pi to 1000 digits is calculated via…

./pi 1000

The ./ in front of the program name is needed if you are executing the program from the directory that you are in, or else Linux will go off searching through the path for programs named pi. If you have created a bin directory under your home folder you can put pi, or any executable code there and run it from other directories on the machine by just calling the program without the ./

Run of Pi program for 1000 digits

Approximation of PI to 1000 digits

3.1415926535  8979323846  2643383279  5028841971  6939937510  5820974944
5923078164  0628620899  8628034825  3421170679  8214808651  3282306647
0938446095  5058223172  5359408128  4811174502  8410270193  8521105559
6446229489  5493038196  4428810975  6659334461  2847564823  3786783165
2712019091  4564856692  3460348610  4543266482  1339360726  0249141273
7245870066  0631558817  4881520920  9628292540  9171536436  7892590360
0113305305  4882046652  1384146951  9415116094  3305727036  5759591953
0921861173  8193261179  3105118548  0744623799  6274956735  1885752724
8912279381  8301194912  9833673362  4406566430  8602139494  6395224737
1907021798  6094370277  0539217176  2931767523  8467481846  7669405132
0005681271  4526356082  7785771342  7577896091  7363717872  1468440901
2249534301  4654958537  1050792279  6892589235  4201995611  2129021960
8640344181  5981362977  4771309960  5187072113  4999999837  2978049951
0597317328  1609631859  5024459455  3469083026  4252230825  3344685035
2619311881  7101000313  7838752886  5875332083  8142061717  7669147303
5982534904  2875546873  1159562863  8823537875  9375195778  1857780532
1712268066  1300192787  6611195909  2164201989

Calculations Completed!
Time: 1 seconds

It is possible to run this program to see how the various machines that you own and compare the CPU performance by using the pi program.

Minimal Boot

More advanced benchmarking of a machine would involve trying to boot using a disk that would allow only the command line and a minimal amount of background stuff to load. I might try this at some point to see what difference it makes. I would try to boot from a Ultimate Boot CD (UBCD) and then go into the mode that loads the minimal Linux boot and somehow get pi.c loaded via USB stick maybe? It would be an experiment!

For more advanced testing that runs outside of the OS it is possible to run the code included on the UBCD, for example the  for a any machine that will boot from CD. The CD contains a suite of benchmarking, testing and stress testing tools, in addition to other tools for working with a hard drive and unlocking machines.

Pi Benchmarking Script

For benchmarking I close out of all applications that are running and open one terminal to execute the pi program.

Code

Below is code for a script file that will run the pi program,looping it multiple times, calculating the results for 1000,2000,4000,8000,16000,32000,64000,128000 and 256000 digits. I wanted to see if there would be any noticeable variations in the time to calculate various amounts of digits among the machines. I didn’t notice much of a deviation when I lined up plots of the various machines. They were all nominally nearly a constant multiple of speeds across the multiple levels of pi calculation

The script creates two temporary files cols.txt that lists the number of digits it has run up to. Additionally a file called results.txt captures the amount of time that it took to calculate the corresponding number of digits.

#! /bin/bash
rm cols.txt
rm results.txt

DIR=/home/erick/pi
x=1000


while [ $x -le 300000 ]; do

echo $x
echo $x >> cols.txt

$DIR/pi $x > temp.txt
tail -n 1 temp.txt >> results.txt

x=$(( $x * 2 ))

done;

Results of the Pi Benchmarking Script

Raspberry Pi Model B, single core CPU at 700MHz
Time: 1 seconds
Time: 1 seconds
Time: 5 seconds
Time: 21 seconds
Time: 86 seconds
Time: 352 seconds
Time: 1438 seconds
Time: 5835 seconds
Time: 23507 seconds
Dell Dimension 530-mt Xeon dual core processor at 2.4GHz

The machine has 1GB RAM and I don’t think that comes into play running this program. The pi program is only running on one of the cores in this example.

Time: 0 seconds
Time: 0 seconds
Time: 1 seconds
Time: 3 seconds
Time: 14 seconds
Time: 58 seconds
Time: 232 seconds
Time: 932 seconds
Time: 3741 seconds
Dell Dimension 2400: Pentium 4 single core at 2.4GHz

Nearly identical performance as the Dell Dimension. This machine has 1.5GB RAM.

Time: 0 seconds
Time: 0 seconds
Time: 1 seconds
Time: 4 seconds
Time: 13 seconds
Time: 56 seconds
Time: 223 seconds
Time: 898 seconds
Time: 3596 seconds

Suprise

I benchmarked two older laptops, an old Dell Inspiron (2003) running Ubuntu 10.04 and a not so old Toshiba Satellite A135 (2009), Mint 17 xfce, both 1.6GHz processors.

Pentium M on the Dell and Celeron M on the Toshiba. 333 and 533 MHz busses respectively. I thought that they would be fairly similar in performance but was pleasantly surprised to find the Toshiba was a decent amount faster than all of the other machines! This was a machine that had only 512MB RAM and ran Vista poorly and I almost scrapped it. Until I bought another stick of RAM and loaded Linux Mint 17 on it! It is like a miracle how much better it runs. It is a good test machine to try out Mint as I might consider it for a future desktop machine.

Dell Inspiron

Time: 0 seconds
Time: 0 seconds
Time: 1 seconds
Time: 3 seconds
Time: 13 seconds
Time: 54 seconds
Time: 276 seconds
Time: 1134 seconds
Time: 4526 seconds

Toshiba Satellite

Time: 0 seconds
Time: 0 seconds
Time: 0 seconds
Time: 2 seconds
Time: 7 seconds
Time: 30 seconds
Time: 164 seconds
Time: 692 seconds
Time: 2747 seconds

A ratio of 1.82 times faster, not bad at all.

 

 

Disk Benchmarking

Disk benchmarking was done by writing from the /dev/null bitbucket to disk, flushing caches and then reading back a 1GB file, discarding it into /dev/null.

Write Script

The follow code is copied into a script file ending in .sh and made executable using chmod +x filename, will write 1GB of zeros to a file named after the of=.

#!/bin/bash
dd if=/dev/zero of=/home/erick/testfile-1024x1M bs=1M count=1024

Read Script

The following code is copied into a script file ending in .sh and made executable using chmod +x filename. This script will read back the file created by the write script, dumping it into the null device. It will read it back in 8k blocks.

#!/bin/bash
dd if=/home/erick/testfile-1024x1M of=/dev/null bs=8k

Flush Caches

In order to test the disk if the RAM on the the machine is sufficiently large and the write operation leaves the written data it in the cache, you need to flush the cache in order to have the machine actually read it from disk. Linux is pretty clever about using RAM that is not doing anything, not already in use for programs and OS, for a disk read/write cache. I remember in the old days of DOS that there were a number of these utility programs that could be loaded that would use some RAM as a cache, effectively Linux does the same thing natively.

Take the following code, copy into a file ending in .sh, such as flush-caches.sh and make executable via the chmod +x filename command and then run it between writing and reading the disk.

#!/bin/bash
sudo sh -c "sync && echo 3 > /proc/sys/vm/drop_caches"

Example Results

These disk write and read utilities can be used to test harddrives, USB sticks, SD cards, RAM Drives (effectively tests RAM speed) and so forth. It even can be used to test the network speed when a drive is mounted using NFS ( or using rsync, scp or sftp ) as this will usually be the speed constraint and not the drive R/W speed.

Dell Dimension 530-mt Primary Hard Drive

Write

erick@Precision-WorkStation-530-MT:~/bin$ ./harddrive-write-test.sh
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 24.0374 s, 44.7 MB/s

Read

erick@Precision-WorkStation-530-MT:~/bin$ ./harddrive-read-test-8k.sh

131072+0 records in

131072+0 records out

1073741824 bytes (1.1 GB) copied, 25.3427 s, 42.4 MB/s

Toshiba Satellite A135

Write

erick@erick-Satellite-A135 ~/bin $ ./harddrive-write-test.sh
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 36.6714 s, 29.3 MB/s

Read

erick@erick-Satellite-A135 ~/bin $ ./harddrive-read-test-8k.sh
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB) copied, 35.093 s, 30.6 MB/s

 

Old, circa 1998 machine, 4GB primary Hard Drive (/dev/sda) and 8GB Compact Flash card (/dev/sdb) Secondary drive

The interesting thing here besides just how slow the speeds are, is the fact that the CF card is actually faster than the hard drive. The drive is actually pretty loud on this machine as well. When CRON runs at the top of the hour, there is definitely a small burst of sound, enough to serve as a reminder of the time if you are in the same room as the machine. When you SSH into it, you can hear it grind away for about 2 seconds as it reconciles the password! I used this machine for remote monitoring before I had a Raspberry Pi, works good enough for that with it’s limited RAM and HD space. Now it is just a backup in case the Pi is down.

Disk /dev/sda: 4303 MB, 4303272960 bytes

Write: 1073741824 bytes (1.1 GB) copied, 144.733 s, 7.4 MB/s

Read: 1073741824 bytes (1.1 GB) copied, 138.498 s, 7.8 MB/s

Disk /dev/sdb: 8195 MB, 8195604480 bytes

Write: 1073741824 bytes (1.1 GB) copied, 91.4438 s, 11.7 MB/s

Read: 1073741824 bytes (1.1 GB) copied, 73.3132 s, 14.6 MB/s

Raspberry Pi SD Card

Using a shorter test via the following script to write…

#!/bin/bash
dd if=/dev/zero of=/home/erick/testfile-10000x8k bs=8k count=10000

and to read…

#!/bin/bash
dd if=/home/erick/testfile-10000x8k of=/dev/null bs=8k
Write To SD Card
erick@raspberrypi ~/bin $ ./sd-write-test.sh
10000+0 records in
10000+0 records out
81920000 bytes (82 MB) copied, 8.45663 s, 9.7 MB/s
Read Back from SD Card
erick@raspberrypi ~/bin $ ./sd-read-test-8k.sh
 10000+0 records in
 10000+0 records out
 81920000 bytes (82 MB) copied, 4.38749 s, 18.7 MB/s
RAM Drive Performance

tmp on my Raspberry Pi is set up as a RAM Drive (RAMFS) on my Raspberry Pi. So this gives some indication of how fast the RAM can be wrote and read to.

Using the following write script…

#!/bin/bash
dd if=/dev/zero of=/tmp/testfile-1000x8k bs=8k count=1000

and read script…

#!/bin/bash
dd if=/tmp/testfile-1000x8k of=/dev/null bs=8k
Write to RAMFS
erick@raspberrypi ~/bin $ ./ram-write-test.sh
1000+0 records in
1000+0 records out
8192000 bytes (8.2 MB) copied, 0.0458338 s, 179 MB/s
Read from RAMFS
erick@raspberrypi ~/bin $ ./ram-read-test-8k.sh
1000+0 records in
1000+0 records out
8192000 bytes (8.2 MB) copied, 0.0339184 s, 242 MB/s

 

 

end

CPU Temperature Monitoring on the Raspberry Pi

Linux, Raspberry Pi, , , , , , , ,

One of my thoughts for using the Raspberry Pi is to monitor ambient temperature in my house and send me warnings if it is too low. This would be helpful when away from home in the winter and there is an issue with the heat that I might need to know about fairly quickly.

Eventually I might consider building in a weather station capability into the Raspberry Pi by using both an indoor temperature, humidity and barometric sensor and an outdoor temperature and humidity sensor. So with that in mind I also want to make some simple graphs of data every hour or so. Something that can be just saved into a text file and requires no extra graphing code on the client or the server.

CPU Temperature Experiment

I started out by experimenting with taking reading of the CPU temperature of the Raspberry Pi in order to get a little practice with some live data while I was awaiting the arrival of a Bosch BME280 ambient temp., humidity and barometric pressure sensor.

sSMTP

I plan on using sSMTP to send me warnings, such as when my ambient temperature falls below a critical value. I have already worked up some code to monitor the CPU temperature and email me if it goes out of a specific range, to get a bit of practice with coding this into a script. Plus it was a good way to try out sSMTP. sSMTP will be covered in more detail in a separate post.

Snippet of script that sends email if the upper and lower CPU temp limits are exceeded

The temperature read by the cat /sys/class/thermal/thermal_zone0/temp is in milliCelsius, so using cut on it will grab whole degree values. The rest of the code is pretty straightforward. The variables minimum and maximum are setup to be the appropriate values in whole degrees Celsius. mailaddr is exactly what it sounds like. What is nice is that it just gives you a very simple method of sending a basic email warning when the temperatures get out of bounds.

# Read the temp and cut it to grab leftmost 2 characters, integer Temp
temp="`cat /sys/class/thermal/thermal_zone0/temp | cut -c1-2`"

#echo $temp

# Mail if about or below the limits
if (( $temp > $maximum )); then
   #echo "above"
   echo "Rasp Pi CPU Temp = $temp. " | mail -s "Rasp Pi HIGH CPU Temp > $maximum" $mailaddr
elif (( $temp < $minimum )); then
   #echo "below"
   echo "Rasp Pi CPU Temp = $temp. " | mail -s "Rasp Pi LOW CPU Temp < $minimum" $mailaddr

fi

 

Simple Graphical Temperature Logging

To just monitor the CPU temperature on a hourly basis and have a simple graphical representation of it I had to do some digging online.

I dug around the web and found a bit of Perl code to make a simple ASCII graph of the temperatures that I was measuring for the CPU. I wanted to be able to simply post to the web a running capture of temperatures, grabbed hourly by using CRON. I did not want to have to rely on graphing tools either on the Raspberry Pi or the client computer to interpret data. I spent a while searching for something simple and this was about as easy as it can get. I found a one line, long line! Method for creating a simple graph using Perl. I looked around a lot on line and wanted to avoid anything that would have to be installed on either the Raspberry Pi or the client machine to be able to plot data. So after a long search this piece of Perl code is about as simple as it gets and will plot and ASCII based graph across one line each time it executes.

The key variables in the code are min, max and w. The variable w controls the width of the plot in characters that it will be allowed to reach when the input value is at max. The variables min and max control the minimum and maximum input values expected and scale the graph accordingly. PLOTTABLE_FILE is the path/filename of the file that the output will be sent to.

 PLOTTABLE_FILE=/var/www/tmp/cputemp-milli-celsius.txt

Perl Code Snippet

The code resides in a script in the /etc/cron.hourly directory and produces a file that is readable via the web, with a new entry every hour. The filename is cputemp, not the lack of a .sh extension. This is important when you create scripts that will reside in the “CRON” folders. It also has to be made executable by using sudo chmod +x filename.

(cat /sys/class/thermal/thermal_zone0/temp ) | perl -ne '$min=20000; $max=65000; $w=79; use POSIX; $d=ceil(log($max)/log(10)); $w-=$d; $v=$_<$min?0:$_>$max?$max:$_; $s=$w*$v/($max-$min); $bar=join("", ("*")x$s); $bar=~s/.$/|/ if $v==$max; print sprintf("%${d}d ",$_)."$bar\n";' >> $PLOTTABLE_FILE

Sample Output

Below of a capture of a few lines of output from the text file that is generated by the Perl code. It does not look as clean as it would by looking at the text file. So I have a capture of that here…. cputemp-milli-celsius

42236 *********************************************************************
41160 *******************************************************************
41160 *******************************************************************
41160 *******************************************************************
40084 *****************************************************************
41160 *******************************************************************
42774 **********************************************************************
44388 ************************************************************************
45464 **************************************************************************
45464 **************************************************************************
46002 ***************************************************************************

Alternate Method to Read CPU Temperture for the Raspberry Pi

There is another way to read the CPU in a more human readable form. Executing the line below…

/opt/vc/bin/vcgencmd measure_temp

…will show a result like this…

temp=45.5'C

Resources

Source of Perl Script

 

Raspberry Pi

Reduce writes to the Raspberry Pi SD card

Linux, Raspberry Pi, Raspberry Pi Server, , , , , , , , , , ,

After 5 months of solid up-time for my Raspberry Pi server, which has been running great. It has been taking a picture every hour and from them creating a creating a timelapse video every day. Also it is being used as a place to drop files to periodically from other place on the network, a little bit of file storage. Eventually I will add more storage space to it to use it even more for network storage.

Recently, I started to think about the potential wear of the SD card as I came across several articles online dealing with the topic. I decided to make a few changes to the Raspberry Pi configuration to reduce the amount of writing to the SD card.

Write Saving #1: Using a tmpfs

I editted /etc/default/tmpfs. In it the comments state  that /run, /run/lock and /run/shm are already mounted as tmpfs on the Pi by default. Which I have observed. This was a change made a while ago for the Pi according to the buzz online. I additionally set RAMTMP=Yes to add /tmp to the directories put on the tmpfs. This sets up access to /tmp with rwx-rwx-rwx permissions. There was a suggestion that I saw online to limit the sizes of the various directories, I added that as well.

# These were recommended by http://raspberrypi.stackexchange.com/questions/169/how-can-i-extend-the-life-of-my-sd-card
# 07262015, mods for using less of the SD card, RAM optimization.
TMPFS_SIZE=10%VM
RUN_SIZE=10M
LOCK_SIZE=5M
SHM_SIZE=10M
TMP_SIZE=25M

The OS and some programs will use /tmp. But so do I. I created a /tmp/web folder under it when the Raspberry Pi boots. Into this folder files go such as the hourly photo and the daily video that scripts create for the webcam that is attached. I have reduced 3 hourly writes to just one photo. I keep only one on the SD card as I don’t want to risk losing a bunch of them taken during the day if I totally relied on the tmpfs. If I was using a UPS, I would have no problem saving all of them on the tmpfs and occasionally backing up to the SD or another device. The big saver is the daily timelapse.avi for the web that is created daily from all of the hourly captured photos. It is many megs in size gets written daily and it doesn’t matter if I lose it. It can be recreated from the photos at will. So it is the perfect kind of file to throw on a RAM file system.

I also store the hourly and daily logs that I create using the cron driven logcreate script that I run. The logcreate script creates an hourly log that is concatenated into a daily log on the tmpfs then every day the daily log will concatenate into a full log, that is rotated, on the SD card, so I have a permanent record. Need to put the link for this here!

What is a tmpfs?

It is a RAM Disk, a.k.a. RAM Drive that allows RAM to be used as a hard drive. Obviously when the power goes out, it goes away. So we don’t want anything important to go there. But for things like files that I make such as the hourly photos that my Web Cam takes and the video it makes daily and logs, it is perfectly fine for usage. It is not a really big deal if the power went out and I lost this information as it will be recreated shortly anyways.

Caution

The only issue that I see with having logs on a tmpfs would be a situation where the Pi got in a state of weirdness where it started rebooting itself and then you had no logs to track down the problem. Then I suppose, it would be just a matter of changing the /etc/fstab file to revert to putting the logs back onto the SD card for a while to track down the problem. But, for a Raspberry Pi like mine that is running stable and I am not doing many experiments with right now, having the logs in volatile memory is not something I worry about. Plus it is easy to make a script to backup the logs to the SD card or another computer, if you manually reboot it, so you can save them if you like when you have control of the reboots.

Write Saving #2: Turning off swap

If the Raspberry Pi runs out of RAM, not likely if it is a server set up for light duty usage, it will start to use swap which is on the SD card, causing writes to the swap file. Mine rarely touches swap. I would rather tune the thing for better memory use than have it use swap.

It is possible to turn off swap usage using the command…

sudo dphys-swapfile swapoff

This is not persistant and needs to be done on every boot. It could be put into the root crontab by editting it using sudo crontab -e and adding the line. Or creating a script for it along with other items that are to be run at startup.

@boot dphys-swapfile swapoff

Online, people said that there was another way to turn it off by reducing the swap file size to zero, a config file for swap, can’t remember the name. But it is claimed that when it reboots it just overrides that and makes a default 100M swap file.

Write Saving #3: Moving /var/log to a tmpfs

One of the biggest offenders as far as writing to files periodically is the logs that live under /var/log and it’s sub-directories. You can create an entry in /etc/fstab that will create a tmpfs for /var/log. The only caution here is daemons, like Apache that require a directory to exist under /var/log or else they will not start. Apt also has a directory under /var/log, but it creates itself when apt runs for the first time so that is no problem. The apt directory has logs that keep track of what apt installs or uninstalls, good info to know about. News seems to work fine creating a directory for itself too. So for me only Apache is a problem.

  1.  Put an entry in /etc/fstab… 
     tmpfs /var/log tmpfs defaults,noatime,mode=0755 0 0
  2.  Found out that news and apt folders create themselves when these things run.
  3. Apache is the one thing that does not like a missing folder so made a Kludge for now using ~/bin/setup-tmp.sh where I create /var/log/apache2 and chmod it 750. Then I restart apache using apachehup.sh, which just restarts it. Apache was failing to load when I pointed the log dir to /tmp in /etc/apache2/envvars under the export APACHE_LOG_DIR directive.

Write Saving #4: noatime

As you can see above one of the options used in the /etc/fstab file is the noatime option. By default the Raspberry Pi uses this option for the mount of the SD card. If you add mount points of your own to the card, make sure noatime is used. Without it Linux makes a small write each time a file is read to keep track of when it was last accessed, this obviously causes writes. It is possible to use it for the writes to the tmpfs as I am doing above. It saves a bit of time as the system does not have to do a write when a file is just being read.

Another good use of noatime is for drives connected across the network. For example on NFS mounts noatime is a really good choice. The network is generally slower than devices attached to a PC and having to send a write across every time a file is read, slows things down a bit when moving many files.

 

Been running this setup with the RAM savings for a few months now with no problems. I hardly ever see the ACT light blinking on the Pi anymore.

 

The LEDs have the following meanings :

  • ACT – D5 (Green) – SD Card Access
  • PWR – D6 (Red) – 3.3 V Power is present
  • FDX – D7 (Green) – Full Duplex (LAN) connected
  • LNK – D8(Green) – Link/Activity (LAN)
  • 100 – D9(Yellow) – 100Mbit (LAN) connected

From http://www.raspberrypi-spy.co.uk/2013/02/raspberry-pi-status-leds-explained/

FTP on Raspberry Pi. An easy way to make shared folders

Linux, Raspberry Pi, Raspberry Pi Server, Server, , , , , , , , ,

The idea with FTP is to have folders that can be reachable between Linux and Windows, locally and remotely and easily. FTP is not secure, but it can be made secure, that info can be found on the web. For now I am covering the basics of FTP here.

For most things that I need to do, I don’t need the files to be secure anyways, 90% of the time nothing critical is going back and forth across remotely. If it is I would use a secure method of sending files via SSH via SFTP or an SSHFS.

FTP is an old protocol but it just plain works and is compatible with Windows, Linux and Mac. I have tried WebDAV in the past but it is compatible to only a degree with various Windows operating systems. I have had a hard time getting it working correctly on versions of Windows beyond XP, resorting in installing patches to Windows and etc. Generally not easy to implement.

I was also looking at FTP as a native tool typical of server installs. I have experimented with cloud setups such as OwnCloud and Sparkleshare, but with FTP I was looking for something simple and quick to setup, no special software, no mySQL databases running on the Raspberry Pi, no special software on client PCs, that sort of thing.

vsFTP

sudo apt-get install vsftpd

Edit the configuration file

Back it up first then do an edit.

sudo cp /etc/vsftpd.conf /etc/vsftpd.orig
sudo nano /etc/vsftpd.conf

uncomment local_enable = YES

uncomment write_enable = YES

Find this and check that it is set this way…

local_umask=022

Enabling PASV

I have read online that enabling the PASV capability for FTP is a good idea. Frequently when I have FTP’d to various ISP’s sites I have seen them operate in PASV mode. So it stands to reason that if the pro’s are have it set up that way it may have it’s advantages.

Add the following lines to the /etc/vsftp.conf file.

pasv_enable= Yes
pasv_min_port=40000
pasv_max_port=40100

There is nothing magic about the numbers of the port range other than they should be unused by anything else that your setup might require and generally I have seen high numbers used commonly. To work out side of your local network you must enable port forwarding of the range of port numbers through your router configuration.

Changes to vsFTP

With the newer versions of vsFTP there is a change that has occurred since I wrote my previous post about vsFTP (  http://oils-of-life.com/blog/linux/server/additional-utilities-for-a-linux-server/ )

The change has to do with the fact that the root directory of the user has to be non-writable and I have read online that it is best to make it owned by root as well. This is covered below, after the section on adding a user. You need to have a user first before modifying their permissions!

FTP User

To create an FTP user, create it in a way that it does not have a login shell. So that someone who can log in to the FTP account can’t execute shell commands. The line /sbin/nologin may not be in the /etc/shell file and in that case it needs to be added in there. The user basically has to be jailed in their directory and has to have no login shell.

sudo useradd -m -s /sbin/nologin -d /home/user user

I added Documents, public_html directories to the /home/user as well. Then made the users root folder /home/user, owned by root and nonwritable.

cd /home/user
chown user:user Documents
chown user:user public_html

chown root:root /home/user
Make Root of user non writable
sudo chmod a-w /home/user

FTPing on the PC

Now that ftp is set up on the server you will want to be able to connect to it!

Options for connecting…

Command Line, WIndows and Linux

ftp yoursite.com

That gets you into FTP via the command line. The command prompt will now start with ftp> ,that is how you know that you are within the ftp command shell.

It is archaic, but worth knowing when you have to stick a file up or pull it down right at the command line. The commands the ftp prompt accepts are basic, but good enough to get most work done. Type help at the prompt to get a list of commands.

Via Folders

Linux

Just enter the location of the ftp server right into the top of the directory folder and you will be prompted for a password and taken there.

Windows
Windows7/Vista:
  1. Open Computer by clicking the “Start” button, and then clicking Computer.
  2. Right-click anywhere in the folder, and then click Add a Network Location.
  3. In the wizard, select Choose a custom network location, and then click Next.
  4. To use a name and password, clear the Log on anonymously check box.

From: https://www.google.com/search?q=connect+to+ftp+windows+7&ie=utf-8&oe=utf-8

 

 

Kombucha with SCOBY

Kombucha SCOBY Timelapse Video

Fermentation, Raspberry Pi, Time Lapse Photography, , , , , ,

I ran a webcam pointed at a fermentation of Kombucha capturing frames using fswebcam running on a Raspberry Pi server. It ran for a few weeks.

You can see the Kombucha SCOBY’s forming in the video. The setup is behind the PC monitor with a desk lamp and camera pointing towards the back of the desk. This gives fairly steady ambient lighting. I grew a dozen SCOBY’s for the Summer Fermentation Class that was held on June 29th 2015.

SCOBY Timelapse Video AVI Format 8 frames/sec

The video should be viewable most of the time unless I am servicing the Raspberry Pi or I pulled the plug on it during a nasty storm.

Tux Favicons

Linux, Raspberry Pi, , , , ,

While I was setting up and testing one of my servers I noticed that when I access it using Opera on a Kindle, Opera tries to grab a favicon.ico file, which was missing. I noticed the file was missing when I was browsing through the Apache error log on the server. In the log there is a missing file complaint about the favicon.ico whenever I browsed it using Opera.

So I found some graphics of Tux the Penguin and the Raspberry Pi raspberry and made up some quick favicons for the web pages on the Raspberry Pi Server and my main server.

I have loaded them up here in large format and the favicons as well. All the Tux pics come from Linux 2.0 Penguins, see this page for the original art. I used GIMP to edit them which worked well.

Two 300 x 300 pixel examples of the Favicons

Tux Server Favicon
Tux Server Favicon
Tux Raspberry Pi Favicon
Tux Raspberry Pi Favicon

 

Favicon ICO Files

http://oils-of-life.com/blog/wp-content/uploads/2015/07/tux-raspberry.ico

http://oils-of-life.com/blog/wp-content/uploads/2015/07/tux-favicon-32.ico

http://oils-of-life.com/blog/wp-content/uploads/2015/07/tux-favicon-16.ico

http://oils-of-life.com/blog/wp-content/uploads/2015/07/tux-shine-server-big-S-32.ico

Automatic Server Status Page Creation Update

Linux, Raspberry Pi, Raspberry Pi Server, Server, , , , , , , , , , , , , , , , ,

In January 2015 I created a post about automatically creating a status page for a Linux server that I have. Typically this is put under a restricted directory and allows you to see a snapshot of what is happening with the server. I run it by putting the scripts in the /etc/cron.hourly directory on a Linux PC and a Raspberry Pi running Linux.

It serves as a simple way to check up on the server without having to use a tool such as Webmin that requires a login. It also keeps a trail of log files that get rotated on a monthly basis, so there is always a few old ones around to track down any problems and patterns in the operation.

I have found this information useful when I have traced down malfunctions that can occur when setting up a server and also when I was trying to get a webcam up and running and had the USB bus hang up a few times when the cam was overloaded with too much light.

In the new script file I fixed a bug by adding parenthesis around a line that I was trying to echo and I added code to run the w command to show a quick picture on who is logged in, how long the server has been up and running and the values for the average load on the server at the 1, 5 and 15 minute marks.

Logcreate Script

#!/bin/dash
# Remove old log
rm /var/www/status/log.txt
# Print logged outputs into new log.txt
date >> /var/www/status/log.txt
echo >> /var/www/status/log.txt
tail /var/log/syslog >> /var/www/status/log.txt
echo >> /var/www/status/log.txt
free >> /var/www/status/log.txt
echo >> /var/www/status/log.txt
df -h >> /var/www/status/log.txt
echo >> /var/www/status/log.txt
# Top memory using processes http://www.commandlinefu.com/commands/view/3/display-the-top-ten-running-processes-sorted-by-memory-usage
#ps aux | sort -nk +4 | tail >> log.txt
echo "USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND" >> /var/www/status/log.txt
ps aux | sort -nrk 4 | head >> /var/www/status/log.txt
echo >> /var/www/status/log.txt
# Logged in User info using w command
w >> /var/www/status/log.txt
echo >> /var/www/status/log.txt
echo >> /var/www/status/log.txt
# Copy log.txt into the full log that is collected
cat /var/www/status/log.txt >> /var/www/status/fulllog.txt
# Create a free standind copy of the process tree
pstree > /var/www/status/pstree.txt

Alternate Version

I also created a version of the script for a desktop Linux PC that does not have Apache installed.  In it I use a DIR variable to contain the directory that I want the log.txt file stored.

 #!/bin/dash

# User defined variables
# No trailing / on DIR!
DIR=/home/erick/status

# Remove old log
rm $DIR/log.txt
# Print logged outputs into new log.txt
date >> $DIR/log.txt
echo >> $DIR/log.txt
tail /var/log/syslog >> $DIR/log.txt
echo >> $DIR/log.txt
free >> $DIR/log.txt
echo >> $DIR/log.txt
df -h >> $DIR/log.txt
echo >> $DIR/log.txt
# Top memory using processes http://www.commandlinefu.com/commands/view/3/display-the-top-ten-running-processes-sorted-by-memory-usage
#ps aux | sort -nk +4 | tail >> log.txt
echo "USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND" >> $DIR/log.txt
ps aux | sort -nrk 4 | head >> $DIR/log.txt
echo >> $DIR/log.txt
# Logged in User info using w command
w >> $DIR/log.txt
echo >> $DIR/log.txt
echo >> $DIR/log.txt
# Copy log.txt into the full log that is collected
cat $DIR/log.txt >> $DIR/fulllog.txt
# Create a free standing copy of the process tree
pstree > $DIR/pstree.txt

Rotation of Log

In the /etc/cron.monthly directory I have created a file that is called status-log-rotate and it will save backup copies of 2 months worth of the full concatenated server status logs.

#! /bin/bash
DIR=/home/erick/status
mv $DIR/fulllog.txt.1 $DIR/fulllog.txt.2
mv $DIR/fulllog.txt $DIR/fulllog.txt.1

Tweaks for Raspberry Pi

For the Raspberry Pi which has an SD card that I am trying to be conscious of writing to often. I have recently made some modifications to put the /tmp folder onto RAM using tmpfs. I create the hourly log underneath a folder there. Daily via a script it cron.hourly it gets concatenated into a daily log which is under a status folder that has restricted access. This gets appended once per day to the fulllog which actually lives on the SD card. The end result, no multiple hourly writes to the log file, just one append to the full log per day. The only downside is if the power drops and then some log entries will be lost for the day.

Logcreate runs from /etc/cron.hourly for Raspberry Pi

#!/bin/dash
# Set DIR, on Pi this is a temp location for log
DIR=/tmp/web

# Set fixed DIR FIXDIR for files that have to be stored on SD card
# Nevermind, just make a daily log and then copy that to the full log daily.
#FIXDIR=/var/www/status

# Remove old log

rm $DIR/log.txt
# Print logged outputs into new log.txt
date >> $DIR/log.txt
echo >> $DIR/log.txt
tail /var/log/syslog >> $DIR/log.txt
echo >> $DIR/log.txt
free >> $DIR/log.txt
echo >> $DIR/log.txt
df -h >> $DIR/log.txt
echo >> $DIR/log.txt
# Top memory using processes http://www.commandlinefu.com/commands/view/3/display-the-top-ten-running-processes-sorted-by-memory-usage
echo "USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND" >> $DIR/log.txt

ps aux | sort -nrk 4 | head >> $DIR/log.txt
echo >> $DIR/log.txt
# Logged in User info using w command
w >> $DIR/log.txt
echo >> $DIR/log.txt
echo >> $DIR/log.txt
# Copy log.txt into the full log that is collected
cat $DIR/log.txt >> $DIR/dailylog.txt
# Create a free standing copy of the process tree
pstree > $DIR/pstree.txt

dailylog-to-fulllog script, runs from /etc/cron.daily

#! /bin/bash

DIR=/tmp/web
FIXDIR=/var/www/status

echo "----------------------------------------------" >> $DIR/dailylog.txt
date >> $DIR/dailylog.txt
echo "----------------------------------------------" >> $DIR/dailylog.txt
cat $DIR/dailylog.txt >> $FIXDIR/fulllog.txt
rm $DIR/dailylog.txt

Logcreate Output from Raspberry Pi

Below is what the logcreate script will output to the log.txt file on a Raspberry Pi that I have running as a web server.

Sun Jul 12 14:17:01 EDT 2015

Jul 12 13:47:51 raspberrypi dhclient: DHCPACK from 192.168.1.1
Jul 12 13:47:52 raspberrypi dhclient: bound to 192.168.1.17 -- renewal in 40673 seconds.
Jul 12 13:59:01 raspberrypi /USR/SBIN/CRON[28010]: (erick) CMD (aplay /opt/sonic-pi/etc/samples/guit_e_fifths.wav)
Jul 12 13:59:07 raspberrypi /USR/SBIN/CRON[28009]: (CRON) info (No MTA installed, discarding output)
Jul 12 14:00:01 raspberrypi /USR/SBIN/CRON[28013]: (erick) CMD (/home/erick/fswebcam/cron-timelapse.sh >> timelapse.log)
Jul 12 14:00:23 raspberrypi /USR/SBIN/CRON[28012]: (CRON) info (No MTA installed, discarding output)
Jul 12 14:01:01 raspberrypi /USR/SBIN/CRON[28022]: (root) CMD (/home/erick/bin/usbreset /dev/bus/usb/001/004)
Jul 12 14:01:02 raspberrypi /USR/SBIN/CRON[28021]: (CRON) info (No MTA installed, discarding output)
Jul 12 14:09:01 raspberrypi /USR/SBIN/CRON[28053]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Jul 12 14:17:01 raspberrypi /USR/SBIN/CRON[28064]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

             total       used       free     shared    buffers     cached
Mem:        445804     424488      21316          0     106768     260516
-/+ buffers/cache:      57204     388600
Swap:       102396          0     102396

Filesystem      Size  Used Avail Use% Mounted on
rootfs          6.3G  3.1G  3.0G  51% /
/dev/root       6.3G  3.1G  3.0G  51% /
devtmpfs        214M     0  214M   0% /dev
tmpfs            44M  240K   44M   1% /run
tmpfs           5.0M  8.0K  5.0M   1% /run/lock
tmpfs            88M     0   88M   0% /run/shm
/dev/mmcblk0p5   60M   19M   41M  32% /boot

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      2071  0.0  3.0  24896 13652 ?        Ss   Jun28   2:24 /usr/sbin/apache2 -k start
www-data 27745  0.0  1.5  25412  7084 ?        S    09:58   0:00 /usr/sbin/apache2 -k start
www-data 27744  0.0  1.5  24960  6760 ?        S    09:58   0:00 /usr/sbin/apache2 -k start
www-data 27743  0.0  1.5  25428  7116 ?        S    09:58   0:00 /usr/sbin/apache2 -k start
www-data 27742  0.0  1.5  25396  7036 ?        S    09:58   0:00 /usr/sbin/apache2 -k start
www-data 27538  0.0  1.5  25396  7032 ?        S    06:25   0:00 /usr/sbin/apache2 -k start
www-data 27502  0.0  1.5  25404  7036 ?        S    06:25   0:00 /usr/sbin/apache2 -k start
www-data 27501  0.0  1.5  25396  7044 ?        S    06:25   0:00 /usr/sbin/apache2 -k start
www-data 27747  0.0  1.3  24936  6188 ?        S    09:58   0:00 /usr/sbin/apache2 -k start
www-data 27746  0.0  1.3  24936  6188 ?        S    09:58   0:00 /usr/sbin/apache2 -k start

 14:17:02 up 14 days, 12:56,  1 user,  load average: 0.00, 0.01, 0.05
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
erick    pts/0    192.168.1.5      14:04   10:39   1.70s  1.70s -bash

Wake on LAN for PC’s via Raspberry Pi

Linux, Raspberry Pi, Raspberry Pi Server, , , , , , , ,

I found a nice PHP script for Wake on LAN. I loaded it on the Raspberry Pi that I have and configured it for my system. The Raspberry Pi runs 24/7, so I can just navigate to a web page that it serves, hit a button and start up one of my machines at home from anywhere. Mostly this is useful for the starting my Linux file server remotely but I do use it to fire off the desktop too.

Right from the read me file for the code…

REMOTE WAKE/SLEEP-ON-LAN SERVER
=========================
This is simple webapp that runs on your Raspberry Pi to turn it into a remotely accessible Wake/Sleep-On-LAN Server. [Follow the detailed tutorial](http://www.jeremyblum.com/2013/07/14/rpi-wol-server/) on my website for instructions on how to get this working, and forwarded through a router. This is very useful when you have high-powered machine that you don’t want to keep on all the time, but that you want to keep remotely accessible for Remote Desktop, SSH, FTP, etc.

http://www.jeremyblum.com/2013/07/14/rpi-wol-server/

Results

It is rare when something works right out of the box. But, this did, I followed Jeremy Blum’s instructions and within a few minutes I had this working. It has a nice drop-down menu where you can select a computer. It pings it to see if it is awake, then you can wake it from anywhere in the world. Once the WOL packet is sent, the application keeps pinging the PC at a defined interval and you can see when it wakes. I have not tried the sleep functionality as I am using it with Linux PC’s and his outline covers Windows machines. I am sure the code could me modified to shut down a Linux PC somehow. Perhaps it automatically SSH’s in and sends a shutdown command, something like that. I have my Linux server set to shutdown automatically so I don’t need this functionality myself.

It is configurable through an easily understood config.php file as well. You can set the computers name and IP address, MAC address, timing between pings, amount of times to ping the machine and etc.

Also see on this site…

Original Wake on LAN via Ubuntu Linux Post

Windows Wake on LAN Post

Simple WebDAV

Linux, Raspberry Pi Server, Server, , , , , ,

WebDAV, the DAV stands for Distributed Authoring and Version. In its simplest form would be a folder that can be accessed from the web that has a username and password to keep the content locked. There are two versions basically, plain and SSL which is secure in that the data that flows in and out of the folder is encrypted as it moves through the web. In this post I am covering the simple non-SSL form for starters.

This post assumes that Apache is installed, if you need to install it do…

sudo apt-get install apache2

Then load the Apache modules for DAV…

sudo a2enmod dav
sudo a2enmod dav_fs

Create a folder for WebDAV

I created a directory at…

/srv/homes/webdav

…the command…

mkdir -p /srv/homes/webdav

…will allow the folders above webdav, such as homes be created if they do not exist.

Edit the Apache default file

The WebDAV folder access is simply controlled by the sites-available/default file. To edit it run…

sudo nano /etc/apache2/sites-available/default

Towards the bottom of the file right above the section that has the ScriptAlias for the /cgi-bin/ directory, I placed the following code…

Alias /webdav  /srv/homes/webdav
<Location /webdav>
 Options Indexes
 DAV On
 AuthType Basic
 AuthName "webdav"
 AuthUserFile /etc/apache2/webdav.password
 Require valid-user
 </Location>

Adding the Password

Use the htpasswd command to add a password to a webdav.password file. it will prompt you for a password. The file will contain hashed passwords which are not readable.

sudo htpasswd -c /etc/apache2/webdav.password username

For an extra level of protection you can change ownership of the file to root with the group of www-data, so no regular users can access the file. Setting the permission to read-write for owner root and read only for the www-data group…

sudo chown root:www-data /etc/apache2/webdav.password
sudo chmod 640 /etc/apache2/webdav.password

Access the Folder

With everything setup the folder will now appear at http://your-url-here.com/webdav, you can browse to it to test it out. You will be prompted for the user-name and password created earlier in the adding the password step.

Further Potential for WebDAV

  • Setup multiple WebDAV folders.
  • Put a web folder on expanded storage on a Raspberry Pi, such as use a bind mount to point to a USB stick plugged into the Pi for extra storage space.
  • It is possible to set up WebDAV with SSL to secure it in a way that the data flowing in and out of the folder will be secured from prying eyes. With my non-SSL WebDAV folder, I don’t put anything up there that is critical or really private data.
  • It is possible to use DAV for support of calendars across devices, something I will explore in the future.
  • There is an app for the iPhone that I have tried that allows easy uploading and downloading to the WebDAV folder. It is easy to drop attachments from email and etc. to the folder for access on a PC.

Resources

https://www.digitalocean.com/community/tutorials/how-to-configure-webdav-access-with-apache-on-ubuntu-12-04 

WebDAV Resources