Linux and Unix Tools page index
sed

^ = represents the begining of the line or of a char string
$ = represents the end of the line or of the char string
Replace all spaces with new lines:
sed -e 's/\s\+/\n/g' Comment out all lines starting with (1) or all the lines in a file at their begining (2):
sed -e '/192/s/^/#/' fichier (1) sed -e 's/^/#/' fichier (2) Replace a eith b in a file and display on standard out:
sed 's/a/b' file Replace a with null, nothing = /g:
sed 's/a//g' /!\ If sed is ised with the -i option then modifications will be applied into the file content!
sed -i 's/a//g' file /000/ matches all lines containing 000, s is for substitution, which will insert a # in the begining of the line ^:
sed /000/s/^/#/

Remove MAC address from a interface config file:
sed -e '/HWADDR=\[0-9][0-9][a-Z][a-Z]:/s///g' interfaces.cfg echo "HWADDR=00:50:56:9a:54:a3" | sed -e "s/HWADDR=[0-Z:]*//" Comment lines with a MAC address:
sed -e /HWADD/s/^/#/ Add values from a file with a new line after each:
echo -e server01'\n'server02'\n'server03'\n'server04 >> file


cut

The cut command allows to display specific zones in a file, i.e.:
cut -c1 /etc/passwd Will show the first column of the file /etc/passwd.
Lets retrieve some IP addresses from iptables -nvL with cut: iptables -nvL > file cat file [...] 0 0 REJECT all -- * * 47.95.1.0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- * * 117.81.166.134 0.0.0.0/0 reject-with icmp-port-unreachable [...] Using cut on the source IPs in the file:
cat file |cut -c 48-66 > file2 Specifications of the cut command. Options & meanings:
-c1-5 Selects columns 1 to 5 -c14- Selects column 14 up to the last one -c1-3,14-18 Selects multiple ranges of columns we can also specify a field separator with -d option. i.e.:
cut -d: -f6 /etc/passwd Will display the sixth field of the /etc/passwd file, of which the field delimiter will be the:.
Retreiving IP from logs, i.e.:
cat /logs/main_access.log.1 |grep -i wp-login.php |grep -v Pingdom | grep -v ciro | grep -v your_ip_address | \ cut -f 1 -d ''-''


grep and advance usages

grep -c Counts occurrencies of a string in a file.
grep -nc2 where n is show the line Number, c is show context with its value number (2 for instance).

Filter lines with GREP

Grep looks up a string in one or may text files and dsiplays on screen the lines searched. The string is represented by a Basic Regular Expression BRE (default) or Extended eRE with the -E option.
Syntax
grep [options] expreg [files...] grep [options] -e expreg1 -e expreg2 [files...] grep [options] -f file_expreg [files...] Main options : -c shows the number of lines found -e specifies multiple regular expressions to look for -E allows to use eRE -f reads RE from a file -F specifies to NOT interpret the search patern as a RE -i allows a search not case sensistive -l shows only the name of the files containing the searched RE -n show number for each line found -q quiet search, results are not shown on screen -v exclude fron search -w the result must match a complete word -x the result must match an entire line Grep in the second file what is not in the first file:
grep -v -f pingdom-ip.list bloc.list grep -v -f pingdom-ip.list bloc.list > tmp Example:
Using a file called depts2012.txt for the examples
Show line containing 85191: $ grep 85191 depts2012.txt 52 85 85191 3 VENDEE Vendée Using eRE and a non case sensitive search: $ grep -E -i '(paris|vosges)' depts2012.txt 75 75056 0 PARIS Paris 88 88160 4 VOSGES Vosges Show lines not starting with the number 9: $ grep -v '^9' depts2012.txt REGION DEP CHEFLIEU TNCC NCC NCCENR 01 01053 5 AIN Ain 67 67482 2 BAS-RHIN Bas-Rhin 68 68066 2 HAUT-RHIN Haut-Rhin etc... Show lines starting with a 0 OR ending with an s: $ grep -e '^0' -e 's$' depts2012.txt 05 05061 4 HAUTES-ALPES Hautes-Alpes 06 06088 4 ALPES-MARITIMES Alpes-Maritimes 14 14118 2 CALVADOS Calvados 25 25056 2 DOUBS Doubs Shows and numbers all the lines starting with 11: $ grep -n '^11' depts2012.txt Search without showing the results on screen. Show the return status of the command to check its state: grep -q '^11' depts2012.txt $ echo $? 0 Only show the number of lines found: $ grep -c -e '^0' -e 's$' depts2012.txt 21 Search for an exact word: $ grep COTE depts2012.txt # Without the -w option, two lines are displayed: 26 21 21231 3 COTE-D'OR Côte-d'Or 53 22 22278 4 COTES-D'ARMOR Côtes-d'Armor $ grep -w COTE depts2012.txt # With -w only the line containing the exact keyword is displayed: 26 21 21231 3 COTE-D'OR Côte-d'Or Not interpreting the search patern as RE: $ grep -F '..' file .. Search the lines containing only digits, using an eRE: $ grep -E -x '[0-9]+' file # With the -x option 2 34 $ grep -E '^[0-9]+$' file # Same result but without the -x option: 2 34 Using a file containing RE to look for: $ cat search # Search lines ending with t t$ # Search lines containing only digits: ^[0-9]+$ The file named search contains the RE: $ grep -E -f search file erytert 2 34 546 Using grep with a pipe to look up for a specific process:
$ ps -ef | grep apache2 root 302 1 0 06:18 ? 00:00:00 /usr/sbin/apache2 -k start www-data 351 302 0 06:18 ? 00:00:00 /usr/sbin/apache2 -k start www-data 352 302 0 06:18 ? 00:00:00 /usr/sbin/apache2 -k start root 3783 3105 0 14:10 pts/0 00:00:00 grep --color=always apache2 The grep command shows up too in the results, we need to use a second pipe to filter it out:
$ ps -ef | grep apache2 | grep -v grep # -v "grep" to filter its name out root 302 1 0 06:18 ? 00:00:00 /usr/sbin/apache2 -k start www-data 351 302 0 06:18 ? 00:00:00 /usr/sbin/apache2 -k start www-data 352 302 0 06:18 ? 00:00:00 /usr/sbin/apache2 -k start Use cases:
grep -r --color=always ERROR * | egrep --color "11:[0-9]{2}:[0-9]{2}" /var/log/* grep -iE --color=always "ERROR|failing" /var/log/error.log Through SSH:
ssh $SERVER 'grep -iE --color=always "ERROR|failing" /var/log/error.log' ssh $SERVER "grep -E --color=always "02:[0-9]{2}:[0-9]{2}" /var/log/error.log" To test and define:
grep "10:[0-9]{2}:[0-9]{2}" grep "200 [0-9]* -

Grep as a remote exec code checking tool

Source: http://resources.infosecinstitute.com/

grep -RPn "(passthru|shell_exec|system|base64_decode|fopen|fclose|eval)" html_prod/ grep -Rn "shell *(" /var/www grep -Rn "tcp *(" /var/www grep -Rn "system *(" /var/www

Backdoor shells commonly use the shell_exec function for executing arbitrary commands.
Aaside from shell_exec function, most PHP backdoor shells also use functions like base64_decode, eval, phpinfo,
system, php_uname, chmod, fopen, fclose, readfile, edoced_46esab, and passthru.
Thus you could also easily grep these functions:
grep -Rn “shell_exec *(” /var/www grep -Rn “base64_decode *(” /var/www grep -Rn “phpinfo *(” /var/www grep -Rn “system *(” /var/www grep -Rn “php_uname *(” /var/www grep -Rn “chmod *(” /var/www grep -Rn “fopen *(” /var/www grep -Rn “fclose *(” /var/www grep -Rn “readfile *(” /var/www grep -Rn “edoced_46esab *(” /var/www grep -Rn “eval *(” /var/www grep -Rn “passthru *(” /var/www


find

find . -type f -exec grep -i "107.168.129.2" {} \; find . -type f -exec grep -iH "" {} \; <== H force to show file in which string is found find / -name "*.txt" -exec grep -i "license" \; find / -name \*.txt Find if string is present in files that are in a directory tree:
find . -type f -exec grep STRING {} \; What is does:
find . => searches from the current working directory
-type f => specifies the type, here f means we search for a file
-exec => we run a command for each file found
grep STRING {} => we search for "STRING" in the files ( {} double curly brackets means we replace the name of found files).
\; => declares the end of the exec option ( \ allows to add another command after the previous ones)

Find and Delete:
find -exec delete find -delete


cron

Listing & Editing Cron Tabs and Cron Jobs:
cron -l cron -e

Example of job definition:
#.---------------- minute (0 - 59) #| .------------- hour (0 - 23) #| | .---------- day of month (1 - 31) #| | | .------- month (1 - 12) OR jan,feb,mar,apr ... #| | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat #| | | | | #* * * * * user-name command to be executed */30 * * * * root /usr/local/bin/checks-all-30-minutes.sh


set

set [--abefhkmnptuvxBCHP] [-o option-name] [arg ...] DESCRIPTION Set or unset values of shell options and positional parameters. Change the value of shell attributes and positional parameters, or display the names and values of shell variables. Options: -a Mark variables which are modified or created for export. -b Notify of job termination immediately. -e Exit immediately if a command exits with a non-zero status. -f Disable file name generation (globbing). -h Remember the location of commands as they are looked up. -k All assignment arguments are placed in the environment for a command, not just those that precede the command name. -m Job control is enabled. -n Read commands but do not execute them. -o option-name -p Turned on whenever the real and effective user ids do not match. Disables processing of the $ENV file and importing of shell functions. Turning this option off causes the effective uid and gid to be set to the real uid and gid. -t Exit after reading and executing one command. -u Treat unset variables as an error when substituting. -v Print shell input lines as they are read. -x Print commands and their arguments as they are executed. -B the shell will perform brace expansion -C If set, disallow existing regular files to be overwritten by redirection of output. -E If set, the ERR trap is inherited by shell functions. -H Enable ! style history substitution. This flag is on by default when the shell is interactive. -P If set, do not follow symbolic links when executing commands such as cd which change the current directory. -T If set, the DEBUG trap is inherited by shell functions. unset a variable: unset TMOUT var1=XXXX echo $var1 var1=XXX unset var1 The variable var1 is now empty


awk

Example:
history | awk '{print $2}' | sort | uniq -c | sort -rn | head -10 var=$(awk -F'=' '{print $2}') awk '{print $n}' // where n = field number Format: awk '{print $field_number}' i.e.: awk '{print $2}' Using variable with awk:
# TEST=test # echo "1 2 3" | awk '{print "'"$TEST"'"$1}' test1

A little more awk, sed and field separator before the end:
Removing commented and empty line from a configuration file before displaying it:
sed '/^#\|^$/d' apache2.conf Changing the field separator IFS to process lines or variables containing a enpty space:
IFS=[new field separator with an empty space] after processing:
unset IFS Example:
IFS=$'\n' # Will only consider the new line as a separator unset IFS


Linux Miscallenous Commands Memo

The command below renames all .pdf files to .doc, here 's/\.pdf$/\.doc/' is the rule:
$ rename -v 's/\.pdf$/\.doc/' *.pdf $ shred -zvu file.pdf

Shred: The options used in the above command:
-z – adds a final overwrite with zeros to hide shredding. -u – helps to truncate and remove file after overwriting. -v – shows progress.

The man command is used to display manual entry pages of commands, when used with the -k switch, it searches the short descriptions and manual page names for the keyword printf (such as adjust, apache and php in the commands below) as regular expression.
man -k apache man -k php

Rclone & Rsync

rclone sync --copy-links DIRECTORY/ google:/DIRECTORY rclone copy ~/DIRECTORY google:/DIRECTORY/ rclone copy google:DIRECTORY/SUB_DIRECTORY ./ sudo rsync -a --delete-after $HOME/ /media/HDD/ rsync -a --delete-after $HOME /mnt/HDD