Category Archives: Scripts

Writing

ChatGPT at it’s finest: writing short helpful code

ChatGPT, Scripts, Trading, , , , , , , , , , , , , ,

ChatGPT at it’s finest: writing short helpful code without having to do fancy work.

The following is a dialog with ChatGPT. I had this script that I would normally have to hand edit all of the ticker  symbols by adding them to the end of the lines of code. I am sure there is some clever way using sed or awk but, I figured why not get ChatGPT to help.

ChatGPT then was also able to generate the Tags for this post,by feeding the post into a prompt and I used Stable Diffusion to generate the featured image using the blog title as the prompt. Sometimes it is good to use ChatGPT to proofread as well but, this post was short and simple enough not to do that.

User

I want to show you an output, in the left column are the ticker symbols for a number of cryptocurrencies, I want you to remember all of them as they will be used in the next question that I will pose…..

Currency  Price      Available  Hold      Avail in USD  Held in USD
AAVE 98.16          0.0 0.0              0.0
ACH 0.020353          0.0 0.0              0.0
ADA 0.3859          0.0 0.0              0.0
ALGO 0.1341          0.0 0.0              0.0
AMP 0.00248          0.0 0.0              0.0
ANKR 0.0244          0.0 0.0              0.0
ATOM 9.283          0.0 0.0              0.0
AVAX 20.66          0.0 0.0              0.0
AXS 6.8          0.0 0.0              0.0
BAL 3.68          0.0 0.0              0.0
BAND 1.417          0.0 0.0              0.0
BAT 0.21159          0.0 0.0              0.0
BCH 223.72          0.0 0.0              0.0
BNT 0.75          0.0 0.0              0.0
BOND 2.2          0.0 0.0              0.0
BTC 37964.39          0.0 0.0              0.0
CGLD 0.53          0.0 0.0              0.0
COMP 51.17          0.0 0.0              0.0
CRV 0.5658          0.0 0.0              0.0
CVC 0.1102          0.0 0.0              0.0
DAI 0.9999          0.0 0.0              0.0
DOGE 0.08033          0.0 0.0              0.0
DOT 5.263          0.0 0.0              0.0
ETC 18.86          0.0 0.0              0.0
ETH 2054.84          0.0 0.0              0.0
FARM 32.73          0.0 0.0              0.0
FET 0.5347          0.0 0.0              0.0
FIL 4.545          0.0 0.0              0.0
GLM 0.2418          0.0 0.0              0.0
GRT 0.1483          0.0 0.0              0.0
ICP 4.605          0.0 0.0              0.0
KNC 0.7065          0.0 0.0              0.0
LINK 14.588          0.0 0.0              0.0
LOOM 0.0673          0.0 0.0              0.0
LPT 6.33          0.0 0.0              0.0
LRC 0.216          0.0 0.0              0.0
LTC 69.78          0.0 0.0              0.0
MANA 0.4405          0.0 0.0              0.0
MATIC 0.7528          0.0 0.0              0.0
MIR 0.0941          0.0 0.0              0.0
MKR 1483.6          0.0 0.0              0.0
MLN 15.12          0.0 0.0              0.0
NKN 0.1064          0.0 0.0              0.0
NMR 15.68          0.0 0.0              0.0
OGN 0.12922          0.0 0.0              0.0
OMG 1.7217          0.0 0.0              0.0
OXT 0.0724          0.0 0.0              0.0
PLA 0.204          0.0 0.0              0.0
POLY 0.1963          0.0 0.0              0.0
QNT 99.56          0.0 0.0              0.0
REN 0.0823          0.0 0.0              0.0
REQ 0.083          0.0 0.0              0.0
RLC 1.4794          0.0 0.0              0.0
RLY 0.0126          0.0 0.0              0.0
SKL 0.0436          0.0 0.0              0.0
SNX 3.392          0.0 0.0              0.0
SOL 58.2          0.0 0.0              0.0
STORJ 0.7658          0.0 0.0              0.0
UMA 1.953          0.0 0.0              0.0
UNI 6.161          0.0 0.0              0.0
USD              0.01 0.0
USDC              0.0 0.0
XLM 0.118572          0.0 0.0              0.0
XTZ 0.817          0.0 0.0              0.0
YFI 8519.94          0.0 0.0              0.0
ZEC 29.03          0.0 0.0              0.0
ZRX 0.39338          0.0 0.0              0.0

I don’t have the ability to remember information from one interaction to the next, but I can certainly help you analyze or work with the data you’ve provided in this session. If you have a specific question or task related to this cryptocurrency data, feel free to let me know, and I’ll do my best to assist you!

User

Can you print out all of the symbols in the left column?

Certainly! Here are all the ticker symbols from the left column:

objectivec
AAVE, ACH, ADA, ALGO, AMP, ANKR, ATOM, AVAX, AXS, BAL, BAND, BAT, BCH, BNT, BOND, BTC, CGLD, COMP, CRV, CVC, DAI, DOGE, DOT, ETC, ETH, FARM, FET, FIL, GLM, GRT, ICP, KNC, LINK, LOOM, LPT, LRC, LTC, MANA, MATIC, MIR, MKR, MLN, NKN, NMR, OGN, OMG, OXT, PLA, POLY, QNT, REN, REQ, RLC, RLY, SKL, SNX, SOL, STORJ, UMA, UNI, USD, USDC, XLM, XTZ, YFI, ZEC, ZRX

If you have any specific tasks or questions related to these symbols, feel free to let me know!

User

OK, good, you remembered them. Now for the next step. I am going to show an example using btc and I would like you to create a listing with all of the ticker symbols from the left column. Example: $DIR/rsi-fractals-avgs-bbands-txt-summary.py btc

Sure, I understand. If you could provide an example of the desired output or clarify the specific format or information you’re looking for related to each ticker symbol (similar to the example you provided with btc), I’d be happy to assist you further.

User

I am looking for you to create code that has $DIR/rsi-fractals-avgs-bbands-txt-summary.py on every line but with the ticker symbols from the left column appear on that line right after it.

Certainly! It sounds like you want a list of commands where $DIR/rsi-fractals-avgs-bbands-txt-summary.py is followed by each ticker symbol. Here’s an example Python script that generates those commands:

python
# List of ticker symbols ticker_symbols = [     "AAVE", "ACH", "ADA", "ALGO", "AMP", "ANKR", "ATOM", "AVAX", "AXS", "BAL",     "BAND", "BAT", "BCH", "BNT", "BOND", "BTC", "CGLD", "COMP", "CRV", "CVC",     "DAI", "DOGE", "DOT", "ETC", "ETH", "FARM", "FET", "FIL", "GLM", "GRT", "ICP",     "KNC", "LINK", "LOOM", "LPT", "LRC", "LTC", "MANA", "MATIC", "MIR", "MKR",     "MLN", "NKN", "NMR", "OGN", "OMG", "OXT", "PLA", "POLY", "QNT", "REN", "REQ",     "RLC", "RLY", "SKL", "SNX", "SOL", "STORJ", "UMA", "UNI", "USD", "USDC", "XLM",     "XTZ", "YFI", "ZEC", "ZRX" ] # Directory variable DIR = "$DIR" # Generate commands commands = [f"{DIR}/rsi-fractals-avgs-bbands-txt-summary.py {symbol}" for symbol in ticker_symbols] # Print commands for command in commands:     
       print(command)

Replace $DIR with the actual directory you want to use. This script will generate commands for each ticker symbol, following the format you provided. If you have any specific adjustments or requirements, feel free to let me know!

OK how did it turn out?

This is the output, just as I expected and was looking for. Saved me probably about 15 minutes of tedious hand editing…

$DIR/rsi-fractals-avgs-bbands-txt-summary.py AAVE
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ACH
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ADA
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ALGO
$DIR/rsi-fractals-avgs-bbands-txt-summary.py AMP
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ANKR
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ATOM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py AVAX
$DIR/rsi-fractals-avgs-bbands-txt-summary.py AXS
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BAL
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BAND
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BAT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BCH
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BNT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BOND
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BTC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py CGLD
$DIR/rsi-fractals-avgs-bbands-txt-summary.py COMP
$DIR/rsi-fractals-avgs-bbands-txt-summary.py CRV
$DIR/rsi-fractals-avgs-bbands-txt-summary.py CVC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py DAI
$DIR/rsi-fractals-avgs-bbands-txt-summary.py DOGE
$DIR/rsi-fractals-avgs-bbands-txt-summary.py DOT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ETC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ETH
$DIR/rsi-fractals-avgs-bbands-txt-summary.py FARM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py FET
$DIR/rsi-fractals-avgs-bbands-txt-summary.py FIL
$DIR/rsi-fractals-avgs-bbands-txt-summary.py GLM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py GRT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ICP
$DIR/rsi-fractals-avgs-bbands-txt-summary.py KNC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LINK
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LOOM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LPT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LRC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LTC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MANA
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MATIC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MIR
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MKR
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MLN
$DIR/rsi-fractals-avgs-bbands-txt-summary.py NKN
$DIR/rsi-fractals-avgs-bbands-txt-summary.py NMR
$DIR/rsi-fractals-avgs-bbands-txt-summary.py OGN
$DIR/rsi-fractals-avgs-bbands-txt-summary.py OMG
$DIR/rsi-fractals-avgs-bbands-txt-summary.py OXT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py PLA
$DIR/rsi-fractals-avgs-bbands-txt-summary.py POLY
$DIR/rsi-fractals-avgs-bbands-txt-summary.py QNT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py REN
$DIR/rsi-fractals-avgs-bbands-txt-summary.py REQ
$DIR/rsi-fractals-avgs-bbands-txt-summary.py RLC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py RLY
$DIR/rsi-fractals-avgs-bbands-txt-summary.py SKL
$DIR/rsi-fractals-avgs-bbands-txt-summary.py SNX
$DIR/rsi-fractals-avgs-bbands-txt-summary.py SOL
$DIR/rsi-fractals-avgs-bbands-txt-summary.py STORJ
$DIR/rsi-fractals-avgs-bbands-txt-summary.py UMA
$DIR/rsi-fractals-avgs-bbands-txt-summary.py UNI
$DIR/rsi-fractals-avgs-bbands-txt-summary.py USD
$DIR/rsi-fractals-avgs-bbands-txt-summary.py USDC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py XLM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py XTZ
$DIR/rsi-fractals-avgs-bbands-txt-summary.py YFI
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ZEC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ZRX

What’s it for?

It is screener code that will go into a script that prints out all the cryptos in the list sorted by their position relative to their 20 day moving average. It uses the same logic as used for the Mayer Multiple, except it is on a 20 day MA basis.

X = Current Price / 20DMA

Code

#!/bin/bash 
DIR="/home/erick/python/ta"


rm $DIR/output-rsi-er-price-sma20-multiple.csv

$DIR/rsi-fractals-avgs-bbands-txt-summary.py AAVE
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ACH
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ADA
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ALGO
$DIR/rsi-fractals-avgs-bbands-txt-summary.py AMP
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ANKR
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ATOM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py AVAX
$DIR/rsi-fractals-avgs-bbands-txt-summary.py AXS
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BAL
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BAND
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BAT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BCH
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BNT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BOND
$DIR/rsi-fractals-avgs-bbands-txt-summary.py BTC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py CGLD
$DIR/rsi-fractals-avgs-bbands-txt-summary.py COMP
$DIR/rsi-fractals-avgs-bbands-txt-summary.py CRV
$DIR/rsi-fractals-avgs-bbands-txt-summary.py CVC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py DAI
$DIR/rsi-fractals-avgs-bbands-txt-summary.py DOGE
$DIR/rsi-fractals-avgs-bbands-txt-summary.py DOT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ETC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ETH
$DIR/rsi-fractals-avgs-bbands-txt-summary.py FARM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py FET
$DIR/rsi-fractals-avgs-bbands-txt-summary.py FIL
$DIR/rsi-fractals-avgs-bbands-txt-summary.py GLM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py GRT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ICP
$DIR/rsi-fractals-avgs-bbands-txt-summary.py KNC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LINK
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LOOM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LPT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LRC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py LTC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MANA
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MATIC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MIR
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MKR
$DIR/rsi-fractals-avgs-bbands-txt-summary.py MLN
$DIR/rsi-fractals-avgs-bbands-txt-summary.py NKN
$DIR/rsi-fractals-avgs-bbands-txt-summary.py NMR
$DIR/rsi-fractals-avgs-bbands-txt-summary.py OGN
$DIR/rsi-fractals-avgs-bbands-txt-summary.py OMG
$DIR/rsi-fractals-avgs-bbands-txt-summary.py OXT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py PLA
$DIR/rsi-fractals-avgs-bbands-txt-summary.py POLY
$DIR/rsi-fractals-avgs-bbands-txt-summary.py QNT
$DIR/rsi-fractals-avgs-bbands-txt-summary.py REN
$DIR/rsi-fractals-avgs-bbands-txt-summary.py REQ
$DIR/rsi-fractals-avgs-bbands-txt-summary.py RLC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py RLY
$DIR/rsi-fractals-avgs-bbands-txt-summary.py SKL
$DIR/rsi-fractals-avgs-bbands-txt-summary.py SNX
$DIR/rsi-fractals-avgs-bbands-txt-summary.py SOL
$DIR/rsi-fractals-avgs-bbands-txt-summary.py STORJ
$DIR/rsi-fractals-avgs-bbands-txt-summary.py UMA
$DIR/rsi-fractals-avgs-bbands-txt-summary.py UNI
$DIR/rsi-fractals-avgs-bbands-txt-summary.py USD
$DIR/rsi-fractals-avgs-bbands-txt-summary.py USDC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py XLM
$DIR/rsi-fractals-avgs-bbands-txt-summary.py XTZ
$DIR/rsi-fractals-avgs-bbands-txt-summary.py YFI
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ZEC
$DIR/rsi-fractals-avgs-bbands-txt-summary.py ZRX


echo "Cur  ,Und,Targt,RSI, ER,20Ml,Mayer Mult."


# Sort by desending price/sma20 multiples
sort -nk6 -t, -r $DIR/output-rsi-er-price-sma20-multiple.csv

the script runs the rsi-fractals-avg-bbands-txt-summary code on all the tickers listed and saves output to…

output-rsi-er-price-sma20-multiple.csv

then sorts the output by the 20sma multiple column

 

Results

Best relative performers on top, worst on bottom. This idea comes from the book Fortune’s Formula and appears as a technique used by Princeton-Newport. Basically sell the top and buy the bottom ( deep value…maybe), or alternativly short the top and long the bottom. Or in the case of crypto the bottom ones might be just be duds that are always behind the pack, maybe don’t even bother with them.

Cur  ,Und,Targt,RSI, ER,20Ml,Mayer Mult.
UNI  ,USD,False,73,0.06,1.18,1.31
SKL  ,USD,False,63,0.07,1.1 ,1.59
FET  ,USD,False,67,0.14,1.09,1.97
SNX  ,USD,False,63,0.09,1.07,1.43
GRT  ,USD,False,60,0.05,1.07,1.37
AXS  ,USD,False,56,0.0 ,1.07,1.2
AMP  ,USD,False,63,0.05,1.06,1.17
MKR  ,USD,False,60,0.13,1.05,1.34
AVAX ,USD,False,61,0.07,1.05,1.61
RLC  ,USD,False,61,0.01,1.04,1.19
OGN  ,USD,False,56,0.05,1.03,1.33
UMA  ,USD,False,52,0.07,1.01,1.13
NMR  ,USD,False,55,0.02,1.01,1.14
MANA ,USD,False,48,0.01,1.01,1.18
DOGE ,USD,False,55,0.02,1.01,1.15
BTC  ,USD,False,54,0.14,1.01,1.27
REQ  ,USD,False,52,0.02,1.0 ,1.1
NKN  ,USD,False,59,0.02,1.0 ,1.1
ICP  ,USD,False,51,0.04,1.0 ,1.14
FARM ,USD,False,53,0.04,1.0 ,1.31
ETH  ,USD,False,47,0.04,1.0 ,1.13
DAI  ,USD,False,44,0.01,1.0 ,1.0
ATOM ,USD,False,49,0.05,1.0 ,1.09
ADA  ,USD,False,57,0.01,1.0 ,1.25
AAVE ,USD,False,52,0.08,1.0 ,1.38
STORJ,USD,False,46,0.13,0.99,2.0
SOL  ,USD,False,51,0.19,0.98,2.12
ALGO ,USD,False,47,0.06,0.98,1.13
ACH  ,USD,False,51,0.02,0.98,1.07
ZEC  ,USD,False,43,0.04,0.97,1.0
XLM  ,USD,False,43,0.05,0.97,1.03
LTC  ,USD,False,42,0.03,0.97,0.89
LINK ,USD,False,46,0.17,0.97,1.8
DOT  ,USD,False,45,0.01,0.97,1.08
COMP ,USD,False,49,0.05,0.97,1.11
QNT  ,USD,False,43,0.02,0.96,0.98
PLA  ,USD,False,45,0.02,0.96,1.19
OXT  ,USD,False,47,0.01,0.96,1.13
ETC  ,USD,False,39,0.0 ,0.96,1.07
CRV  ,USD,False,47,0.08,0.96,0.92
BCH  ,USD,False,36,0.1 ,0.96,1.08
LRC  ,USD,False,42,0.08,0.95,0.99
FIL  ,USD,False,41,0.0 ,0.95,1.14
CGLD ,USD,False,44,0.0 ,0.95,1.07
BAL  ,USD,False,38,0.12,0.95,0.88
XTZ  ,USD,False,38,0.03,0.94,1.03
MLN  ,USD,False,36,0.05,0.94,0.9
BAT  ,USD,False,38,0.01,0.94,1.07
BAND ,USD,False,44,0.01,0.94,1.12
ANKR ,USD,False,43,0.01,0.94,1.06
LPT  ,USD,False,41,0.04,0.92,1.13
GLM  ,USD,False,41,0.02,0.92,1.18
CVC  ,USD,False,48,0.05,0.91,1.27
ZRX  ,USD,False,36,0.09,0.9 ,1.68
YFI  ,USD,False,42,0.04,0.9 ,1.32
MATIC,USD,False,28,0.03,0.9 ,1.1
KNC  ,USD,False,27,0.03,0.9 ,1.11
BNT  ,USD,False,32,0.08,0.9 ,1.56
POLY ,USD,False,30,0.01,0.81,1.2

 

 

ChatGPT will create Tags

Finally by posting this post into a prompt you can ask ChatGPT to generate Tags for the post. I used most of them but trimmed a few that were redundant.

laptop

Scraping a WordPress site for text to use with char-rnn

Linux, Scripts, WordPress, , ,

As with anything else in life, it is possible to change nothing but yourself.
The first step toward making change is simply to change yourself.

In this quick example the gist of scraping a WordPress website using Linux and Lynx will be shown. Wget is great at scraping from the web but, I have found out that it does not always work well with WordPress sites.

Grab some text

Step one of the process is to grab some text to work with. In the example I tried I grabbed all of the text of the posts of this blog by scraping it with wget. I also used the text of the US Constitution to see what the tools would do with it as well. Generally the more text, the better the machine learning code will be at generating something interesting.

Scraping the posts

Using the command line web browser lynx in a script I was able to download the text of the posts on this site. Initially I thought to use wget. But, I remembered that wget will do a good job downloading static sites and sometimes will not do so well with ones like this one that is created in WordPress.

There is probably a way to loop this code in bash, and increment a counter for the pages. But, being that this is a one time thing, I opted for a quick approach instead of thinking too hard on making a loop.

#!/bin/bash
lynx -dump -nolist http://erick.heart-centered-living.org/ > my-posts.txt
lynx -dump -nolist http://erick.heart-centered-living.org/page/2/ >> my-posts.txt
lynx -dump -nolist http://erick.heart-centered-living.org/page/3/ >> my-posts.txt
lynx -dump -nolist http://erick.heart-centered-living.org/page/4/ >> my-posts.txt
lynx -dump -nolist http://erick.heart-centered-living.org/page/5/ >> my-posts.txt
lynx -dump -nolist http://erick.heart-centered-living.org/page/6/ >> my-posts.txt
lynx -dump -nolist http://erick.heart-centered-living.org/page/7/ >> my-posts.txt
lynx -dump -nolist http://erick.heart-centered-living.org/page/8/ >> my-posts.txt
lynx -dump -nolist http://erick.heart-centered-living.org/page/9/ >> my-posts.txt

This code will output a file that contains all of the text from the posts on this site. Up to Fall of 2018 when I ran it.

Cloning Linux Mint Setups

Linux, My Thoughts, Scripts, , , , , , , , , , ,

Recently I swapped in an SSD to be the new primary drive on my media center PC which was running Linux Mint 18.0 on the spinning SATA drive.

This post is basically a brief documentation of the basic steps involved in cloning or upgrading/cloning Linux Mint. Most likely this works fine for Ubuntu as well as Debian as they share a common ancestry. There are most likely limits to this scheme. I imagine things would break badly trying to do this across 17.3 and 18 for example. The base on those is a different version of Ubuntu, 14.04 vs 16.04. I might try to do a clone when the next whole number version of Mint comes along. Just pop in a drive that I don’t care about, or do it on a VM, such as Virtualbox for an experiment.

Plans

The plan is to relieve some of the storage duties of the spinning drive which was filling up. Plus a speed increase as the SSD can move data 4x faster than the spinning drive but more importantly with no moving parts the access time is minute in comparison the the spinning drive. Applications open much faster, boot time is cut by 75%, etc. If the machine needs to use swap it won’t grind down to a halt as well with a fast disk. This machine is a bit older, SATA II, but a Solid State Drive (SSD) still makes a big difference.

The idea is to clone over the home folder but exclude large data such as  the ~/Music folder and leave that on the old drive and mount the drive as additional storage and use a symlink to it.

Old Setup 160GB Spinning Drive
Old Setup: 160GB Spinning Drive
New Setup: 80GB Primary SSD
New Setup: 80GB Primary SSD

Goal

The goal of this post’s example is to  do an upgrade to Linux Mint 18.3 from 18, clone over my user settings and reinstall all programs. Over the past year and a half that the machine has been in use there have been quite a few programs that have been installed on this machine. Many of them run from the command line or are libraries related to some of the machine learning code that gets run in the background on the machine. Needless to say it would be very hard to remember them and a lot of little things would be broken.

Step 1: Install Linux Mint from USB Stick or DVD

This step is pretty basic and is covered elsewhere on the web…

Linux Mint,  Ubuntu , Debian

But needless to say you want to create a user that has the same name and User ID (UID) and Group ID (GID) as on the drive that you will be cloning from.

Step 2: Login on the new machine/drive setup kill your home directory and rsync the old one over

Mount the old drive, doing this from the GUI folder is a fine way to do it. Make note of where it mounts. You can always execute df from the command line to find where it mounted as well

It sounds crazy but, it does work. Login, open a terminal and execute…

rm -rf /home/yourusername

Everything is in memory that the OS needs right now to get through the next step, so nothing will crash and this will give you a blank slate to work with.

Next rsync over your home folder from the old drive ( /dev/sda in my case) making sure that you use the archive option. Using the v and h options as well is helpful as well to produce a lot of output in case you have to trace through a problem.

-v : verbose
-a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
# -h : human-readable, output numbers in a human-readable format

Example:

For me it went something like this…

rsync -avh /media/erick/B0B807B9-E4FC-499E-81AD-CDD246817F16/home/erick /home/

Next log out and then back in. Almost like magic everything should look familiar. The wallpaper on the desktop should look like whatever it was on the old setup, fonts and other desktop sizing customizations should be there. Open the browser and it should be like you left it in the old setup. It is almost like you hibernated the old setup and woke it up after teleporting it’s soul into the new drive.

But, wait the software needs attention

Step 3: Bring over the software too, sort of…

More like apt-get install it over is closer to the truth. I tried following a post on how to do this (https://askubuntu.com/questions/25633/how-to-migrate-user-settings-and-data-to-new-machine) but, what was described in it did not lead to success. The suggestion was the following…

oldmachine$ sudo dpkg --get-selections > installedsoftware
newmachine$ sudo dpkg --set-selections < installedsoftware
newmachine$ sudo apt-get --show-upgraded dselect-upgrade

It didn’t work but, it at least did the collection part. So I wound up using the first part…

oldmachine$ sudo dpkg --get-selections > installedsoftware

…and then brute forced an install by doing some grep,rev,cut,rev again on the input file. Which basically flips every line in the file and removes the word “install” which is now at the beginning and backwards then flips it back over line by line.

The next line with the awk command prepends sudo apt-get install to the front of each line and saves the output to reinstall-software.sh

 installedsoftware-to-apt-get-install.sh
 #!/bin/bash
 cat installedsoftware | grep "install" | rev | cut -c 12- | rev > cleaned-installed-software
 awk '{printf "sudo apt-get install "$0"\n"}' cleaned-installed-software > reinstall-software.sh

Run the reinstall-software.sh script and it will do just what it says, install all of the software that was on the old setup. I believe there is an option for apt-get to preanswer Yes when it comes up and asks you the yes or no question about installing. I left it off so that I could review all the larger sized software being loaded. A few times I hit no by accident so had to re-run the script, no big deal.

Reboot is best now to avoid side-effects

Before going much further a reboot is probably in order as so much has changed on the machine.

For me, during the software install process, I was presented with a question about picking LightDM or another X- windows manager. I picked LightDM because I think that is what I had been using. After I was all done, I put the machine in suspend and it had a bit of trouble coming out of it, having a temporary error related to the X-windows manager. A blue screen came up and had a message about removing a temporary file. Just rebooting the machine cleared this up as the /tmp directory is flushed. Apparently this was something that was set before the upgrade, clone and software install process and did not get unset. Other than that I have seen no side effects from the process of upgrade, clone, software install.

Other Items

If you had files configured outside of the home directory such as /etc/hosts, you will obviously have to copy that over. Also, if you have any /etc/cron.hourly,weekly,monthlies that you put on the old machine. Also, it pays to do a dump of crontab’s using crontab -l > crontab-dump.txt on the old setup so they can be reconfigured to the same settings.

Cloning old to new box

This entire process can be used to clone one computer setup to another, old box to new one for example. Which brings us to…

Final Thoughts: Twin Machines

It is entirely possible to keep two machines in sync using the methods outlined above. I have not tried this but I am tempted to test it out at least. What I am thinking of is a laptop and desktop for instance. The desktop with it’s ability to hold multiple drives with ease works nice here It has one drive with the same OS as the “twin” laptop and is setup as multi OS boot. The steps above are executed, cloning the laptop setup and data to the desktop. It is entirely possible to keep cloning the home folder contents back and forth between the two to keep them sync’d. Even the software can be kept in sync using the method used above to re-install it.

It is possible to do this directly between them, both on at the same time. Or, through a server where they both get backed up to. The only caveat is overwriting and deletions. Such as care when using the –delete option with rsync. There is a potential for a race condition of sorts if settings and files get changed and then clobbered by a sync operation. If I were to try this I would start with a one direction sync. One device is the master and the other the slave. Deletions and settings changes get cloned from master to slave automatically only.

Video Conversion Script for ffmpeg

Linux, Scripts, Time Lapse Photography, , , , , ,

Once and a while I have to convert a video made with Cheese, a .webm, or one made by my camera, .mov to an MP4 which takes up the least amount of space (as of 2017) and seems to be supported across a lot of devices.

The script is both a script to run and a reminder as to the syntax that ffmpeg expects as I seem to occasionally forget and wanted a snippet on line as a quick go to for reference.

In this one scaling is applied via the -s option and the bandwidth via the -b option is limited as well. Plus it allows you to choose the filename for the output file.

If ffmpeg -i %1 %1.mp4 is used for example it would take the input file and convert to mp4, tacking on the mp4 extension with no scaling and bandwidth limiting.

#! /bin/bash
#ffmpeg -i input.wmv -s 480x320 -b 1000k output.mp4
ffmpeg -i %1 -s 480x320 -b 1000k %2

Better Yet Do It Batchwise

For example, with a for loop, this code will simply go through the directory and convert all .webm’s to .mp4’s and it is set up to do scaling too if needed using -s hd480. It also keeps the same filename by changes the extension to the appropriate one for the output file.

#!/bin/bash

for a in ./*.webm; do
#  ffmpeg -i "$a" -qscale:a 0 "${a[@]/%webm/mp4}"
  ffmpeg -i "$a" -s hd480 -qscale:a 0 "${a[@]/%webm/mp4}"
done