Monthly Archives: September 2015

Raspberry Pi

Reduce writes to the Raspberry Pi SD card

Linux, Raspberry Pi, Raspberry Pi Server, , , , , , , , , , ,

After 5 months of solid up-time for my Raspberry Pi server, which has been running great. It has been taking a picture every hour and from them creating a creating a timelapse video every day. Also it is being used as a place to drop files to periodically from other place on the network, a little bit of file storage. Eventually I will add more storage space to it to use it even more for network storage.

Recently, I started to think about the potential wear of the SD card as I came across several articles online dealing with the topic. I decided to make a few changes to the Raspberry Pi configuration to reduce the amount of writing to the SD card.

Write Saving #1: Using a tmpfs

I editted /etc/default/tmpfs. In it the comments state  that /run, /run/lock and /run/shm are already mounted as tmpfs on the Pi by default. Which I have observed. This was a change made a while ago for the Pi according to the buzz online. I additionally set RAMTMP=Yes to add /tmp to the directories put on the tmpfs. This sets up access to /tmp with rwx-rwx-rwx permissions. There was a suggestion that I saw online to limit the sizes of the various directories, I added that as well.

# These were recommended by http://raspberrypi.stackexchange.com/questions/169/how-can-i-extend-the-life-of-my-sd-card
# 07262015, mods for using less of the SD card, RAM optimization.
TMPFS_SIZE=10%VM
RUN_SIZE=10M
LOCK_SIZE=5M
SHM_SIZE=10M
TMP_SIZE=25M

The OS and some programs will use /tmp. But so do I. I created a /tmp/web folder under it when the Raspberry Pi boots. Into this folder files go such as the hourly photo and the daily video that scripts create for the webcam that is attached. I have reduced 3 hourly writes to just one photo. I keep only one on the SD card as I don’t want to risk losing a bunch of them taken during the day if I totally relied on the tmpfs. If I was using a UPS, I would have no problem saving all of them on the tmpfs and occasionally backing up to the SD or another device. The big saver is the daily timelapse.avi for the web that is created daily from all of the hourly captured photos. It is many megs in size gets written daily and it doesn’t matter if I lose it. It can be recreated from the photos at will. So it is the perfect kind of file to throw on a RAM file system.

I also store the hourly and daily logs that I create using the cron driven logcreate script that I run. The logcreate script creates an hourly log that is concatenated into a daily log on the tmpfs then every day the daily log will concatenate into a full log, that is rotated, on the SD card, so I have a permanent record. Need to put the link for this here!

What is a tmpfs?

It is a RAM Disk, a.k.a. RAM Drive that allows RAM to be used as a hard drive. Obviously when the power goes out, it goes away. So we don’t want anything important to go there. But for things like files that I make such as the hourly photos that my Web Cam takes and the video it makes daily and logs, it is perfectly fine for usage. It is not a really big deal if the power went out and I lost this information as it will be recreated shortly anyways.

Caution

The only issue that I see with having logs on a tmpfs would be a situation where the Pi got in a state of weirdness where it started rebooting itself and then you had no logs to track down the problem. Then I suppose, it would be just a matter of changing the /etc/fstab file to revert to putting the logs back onto the SD card for a while to track down the problem. But, for a Raspberry Pi like mine that is running stable and I am not doing many experiments with right now, having the logs in volatile memory is not something I worry about. Plus it is easy to make a script to backup the logs to the SD card or another computer, if you manually reboot it, so you can save them if you like when you have control of the reboots.

Write Saving #2: Turning off swap

If the Raspberry Pi runs out of RAM, not likely if it is a server set up for light duty usage, it will start to use swap which is on the SD card, causing writes to the swap file. Mine rarely touches swap. I would rather tune the thing for better memory use than have it use swap.

It is possible to turn off swap usage using the command…

sudo dphys-swapfile swapoff

This is not persistant and needs to be done on every boot. It could be put into the root crontab by editting it using sudo crontab -e and adding the line. Or creating a script for it along with other items that are to be run at startup.

@boot dphys-swapfile swapoff

Online, people said that there was another way to turn it off by reducing the swap file size to zero, a config file for swap, can’t remember the name. But it is claimed that when it reboots it just overrides that and makes a default 100M swap file.

Write Saving #3: Moving /var/log to a tmpfs

One of the biggest offenders as far as writing to files periodically is the logs that live under /var/log and it’s sub-directories. You can create an entry in /etc/fstab that will create a tmpfs for /var/log. The only caution here is daemons, like Apache that require a directory to exist under /var/log or else they will not start. Apt also has a directory under /var/log, but it creates itself when apt runs for the first time so that is no problem. The apt directory has logs that keep track of what apt installs or uninstalls, good info to know about. News seems to work fine creating a directory for itself too. So for me only Apache is a problem.

  1.  Put an entry in /etc/fstab… 
     tmpfs /var/log tmpfs defaults,noatime,mode=0755 0 0
  2.  Found out that news and apt folders create themselves when these things run.
  3. Apache is the one thing that does not like a missing folder so made a Kludge for now using ~/bin/setup-tmp.sh where I create /var/log/apache2 and chmod it 750. Then I restart apache using apachehup.sh, which just restarts it. Apache was failing to load when I pointed the log dir to /tmp in /etc/apache2/envvars under the export APACHE_LOG_DIR directive.

Write Saving #4: noatime

As you can see above one of the options used in the /etc/fstab file is the noatime option. By default the Raspberry Pi uses this option for the mount of the SD card. If you add mount points of your own to the card, make sure noatime is used. Without it Linux makes a small write each time a file is read to keep track of when it was last accessed, this obviously causes writes. It is possible to use it for the writes to the tmpfs as I am doing above. It saves a bit of time as the system does not have to do a write when a file is just being read.

Another good use of noatime is for drives connected across the network. For example on NFS mounts noatime is a really good choice. The network is generally slower than devices attached to a PC and having to send a write across every time a file is read, slows things down a bit when moving many files.

 

Been running this setup with the RAM savings for a few months now with no problems. I hardly ever see the ACT light blinking on the Pi anymore.

 

The LEDs have the following meanings :

  • ACT – D5 (Green) – SD Card Access
  • PWR – D6 (Red) – 3.3 V Power is present
  • FDX – D7 (Green) – Full Duplex (LAN) connected
  • LNK – D8(Green) – Link/Activity (LAN)
  • 100 – D9(Yellow) – 100Mbit (LAN) connected

From http://www.raspberrypi-spy.co.uk/2013/02/raspberry-pi-status-leds-explained/

FTP on Raspberry Pi. An easy way to make shared folders

Linux, Raspberry Pi, Raspberry Pi Server, Server, , , , , , , , ,

The idea with FTP is to have folders that can be reachable between Linux and Windows, locally and remotely and easily. FTP is not secure, but it can be made secure, that info can be found on the web. For now I am covering the basics of FTP here.

For most things that I need to do, I don’t need the files to be secure anyways, 90% of the time nothing critical is going back and forth across remotely. If it is I would use a secure method of sending files via SSH via SFTP or an SSHFS.

FTP is an old protocol but it just plain works and is compatible with Windows, Linux and Mac. I have tried WebDAV in the past but it is compatible to only a degree with various Windows operating systems. I have had a hard time getting it working correctly on versions of Windows beyond XP, resorting in installing patches to Windows and etc. Generally not easy to implement.

I was also looking at FTP as a native tool typical of server installs. I have experimented with cloud setups such as OwnCloud and Sparkleshare, but with FTP I was looking for something simple and quick to setup, no special software, no mySQL databases running on the Raspberry Pi, no special software on client PCs, that sort of thing.

vsFTP

sudo apt-get install vsftpd

Edit the configuration file

Back it up first then do an edit.

sudo cp /etc/vsftpd.conf /etc/vsftpd.orig
sudo nano /etc/vsftpd.conf

uncomment local_enable = YES

uncomment write_enable = YES

Find this and check that it is set this way…

local_umask=022

Enabling PASV

I have read online that enabling the PASV capability for FTP is a good idea. Frequently when I have FTP’d to various ISP’s sites I have seen them operate in PASV mode. So it stands to reason that if the pro’s are have it set up that way it may have it’s advantages.

Add the following lines to the /etc/vsftp.conf file.

pasv_enable= Yes
pasv_min_port=40000
pasv_max_port=40100

There is nothing magic about the numbers of the port range other than they should be unused by anything else that your setup might require and generally I have seen high numbers used commonly. To work out side of your local network you must enable port forwarding of the range of port numbers through your router configuration.

Changes to vsFTP

With the newer versions of vsFTP there is a change that has occurred since I wrote my previous post about vsFTP (  http://oils-of-life.com/blog/linux/server/additional-utilities-for-a-linux-server/ )

The change has to do with the fact that the root directory of the user has to be non-writable and I have read online that it is best to make it owned by root as well. This is covered below, after the section on adding a user. You need to have a user first before modifying their permissions!

FTP User

To create an FTP user, create it in a way that it does not have a login shell. So that someone who can log in to the FTP account can’t execute shell commands. The line /sbin/nologin may not be in the /etc/shell file and in that case it needs to be added in there. The user basically has to be jailed in their directory and has to have no login shell.

sudo useradd -m -s /sbin/nologin -d /home/user user

I added Documents, public_html directories to the /home/user as well. Then made the users root folder /home/user, owned by root and nonwritable.

cd /home/user
chown user:user Documents
chown user:user public_html

chown root:root /home/user
Make Root of user non writable
sudo chmod a-w /home/user

FTPing on the PC

Now that ftp is set up on the server you will want to be able to connect to it!

Options for connecting…

Command Line, WIndows and Linux

ftp yoursite.com

That gets you into FTP via the command line. The command prompt will now start with ftp> ,that is how you know that you are within the ftp command shell.

It is archaic, but worth knowing when you have to stick a file up or pull it down right at the command line. The commands the ftp prompt accepts are basic, but good enough to get most work done. Type help at the prompt to get a list of commands.

Via Folders

Linux

Just enter the location of the ftp server right into the top of the directory folder and you will be prompted for a password and taken there.

Windows
Windows7/Vista:
  1. Open Computer by clicking the “Start” button, and then clicking Computer.
  2. Right-click anywhere in the folder, and then click Add a Network Location.
  3. In the wizard, select Choose a custom network location, and then click Next.
  4. To use a name and password, clear the Log on anonymously check box.

From: https://www.google.com/search?q=connect+to+ftp+windows+7&ie=utf-8&oe=utf-8