Using fields in a csv file as variable in sge array job - awk

Tag: bash,awk,sungridengine

I can use the following bash script to assign a variable ID from each line (first 1000 lines in this case) of the ids file and run it as argument to pythonscript.py.

#!/bin/bash
#$ -t 1:1000
#$ -N thename
#$ -j y

ids=NOBACK/ids.txt
echo "Starting on : $(date)"
echo "Running on node : $(hostname)"
echo "Current directory : $(pwd)"
echo "Current job ID : $JOB_ID"
echo "Current job name : $JOB_NAME"
echo "Task index number : $SGE_TASK_ID"
ID=`awk "NR==$SGE_TASK_ID" {IDS}` 
echo "id is: $ID"
python pythonscript.py --idarg ID
echo "Finished on : $(date)"

But if the file is a csv file and I need to assign multiple variables, how it could be done?

Best How To :

Imagine you have the following csv file (named super.csv):

 name,postcode,dob
 alan,XXXAAA,11/11/55
 bruji,AAAXXX,20/10/88
 ...
 zorri,AXAXAX,01/01/01

and you want to use the first and third fields as arguments in your sungrid engine array job. The following will extract the fields from the lane equals to the $SGE_TASK_ID:

NAME=$(awk -F, -v "line=$SGE_TASK_ID" 'NR==line {print $1}' super.csv)
DOB=$(awk -F, -v "line=$SGE_TASK_ID" 'NR==line {print $3}' super.csv)

I was messing with double and single quotes. The bash will ignore $ var within single quotes and extend variables within double quotes. Here -v is injecting the $SGE_TASK_ID variable to awk scope (within single quotes).

How to append entry the end of a multi-line entry using any of stream editors like sed or awk

linux,bash,awk,sed,sh

Here's a sed version: /^Host_Alias/{ # whenever we match Host_Alias at line start : /\\$/{N;b} # if backslash, append next line and repeat s/$/,host25/ # add the new host to end of line } If you need to add your new host to just one of the host aliases, adjust...

Calling find more than once on the same folder tree

linux,bash,shell,unix,find

Try this: find . -mmin +35 -or -mmin -25 find supports several logical operators (-and, -or, -not). See the OPERATORS section of the man pages for more details. ==================== EDIT: In response to the question about processing the two matches differently, I do not know of a way to do...

linux - running a process and tailing a file simultaneously

bash,shell,tail

I would simply start the tail in background and the python process in foreground. When the python process finishes you can kill the tail, like this: #!/bin/bash touch /tmp/out # Make sure that the file exists tail -f /tmp/out & pid=$! python test.py kill "$pid" ...

Convert AWK command to sqlite query

sql,awk,sqlite3

SQLite is an embedded database, i.e., it is designed to be used together with a 'real' programming language. It might be possible to import that log file into a database file, but the whole point of having a database is to store the data, which is neither a direct goal...

Matching string inside file and returning result

regex,string,bash,shell,grep

Using sqlite3 from bash on OS X seems fairly straightforward (I'm no expert at this, by the way). You will need to find out which table you need. You can do this with an interactive session. I'll show you with the database you suggested: /Users/fredbloggs> sqlite3 ~/Library/Application\ Support/Dock/desktoppicture.db SQLite version...

Assign and use of a variable in the same subshell

bash,scope,subshell

Let's look to the POSIX specification to understand why this behaves as it does, not just in bash but in any compliant shell: 2.10.2, Shell Grammar Rules From rule 7(b), covering cases where an assignment precedes a simple command: If all the characters preceding '=' form a valid name (see...

awk ternay operator, count fs with ,

awk

awk -F, '{ print ( NF ? $NF : $0 ) }' file ...

AWK|BASH, use double FS and ternary operator

bash,awk

You can use this awk command awk -F, '{if (split($1, a, " ") > 2) print $NF; else print}' file This will output rosario escuela estadist, medellin medellin If you want to get rid of the space before $NF, use this awk command awk -F, '{if (split($1, a, " ")...

Extracting columns within a range AWK

unix,awk

You want to test for 0.75-0.8 but wrote code to test for 0.7-0.75 and you forgot to specify what to test in the second part of your condition. Do this: awk '$2 >= 0.75 && $2 <= 0.8' Also note that you want a numeric comparison not a string comparison...

using sed to replace a line with back slashes in a shell script

regex,bash,shell,ssh,sed

You can use it with ssh and heredoc like this: ssh -t -t [email protected]<<'EOF' sed 's~out_prefix=orderid ^2\\\\d\\+ updatemtnotif/~out_prefix=orderid ^2\\\\d\\+ updatemtnotif_fr/~' ~/path/to/file exit EOF PS: It is important to quote the 'EOF' as shown....

Finding the average of a column excluding certain rows using AWK

linux,bash,awk,scripting

Through awk, $ awk '$5!="99999"{sum+=$5}END{print sum}' file 227.5 Explanation: $5!="99999" if 5th column does not contain 99999, then do {sum+=$5} adding the value of 5th column to the variable sum. Likewise it keeps adding the value of 5th column when awk see's the record which satisfies the given condition. Finally...

how to immediately login through ssh?

bash,ssh

SSH logins can be passwordless with the use of key authentication. Arch has a great documentation page on how to set it up, with steps that should work on most ditributions: https://wiki.archlinux.org/index.php/SSH_keys It boils down to these basic steps (details in linked document): Generate keypair Copy public key to authorized...

Python: can't access newly defined environment variables

python,bash,environment-variables

After updating your .bashrc, perform source ~/.bashrc to apply the changes. Also, merge the two BONSAI-related calls into one: export BONSAI=/home/me/Utils/bonsai_v3.2 UPDATE: It was actually an attempt to update the environment for some Eclipse-based IDE. This is a different usecase altogether. It should be described in the Eclipse help. Also,...

Shell script to loop over files with same names but different extensions

linux,bash,shell

anubhava's solution is excellent if, as they do in your example, the extensions sort into the right order. For the more general case, where sorting cannot be relied upon, we can specify the argument order explicitly: for f in *.ext1 do program "$f" "${f%.ext1}.ext2" done This will work even if...

While loop in bash using variable from txt file

linux,bash,rhel

As indicated in the comments, you need to provide "something" to your while loop. The while construct is written in a way that will execute with a condition; if a file is given, it will proceed until the read exhausts. #!/bin/bash file=Sheetone.txt while IFS= read -r line do echo sh...

How to extract first letters of dashed separated words in a bash variable?

linux,string,bash,shell,variables

This isn't the shortest method, but it doesn't require any external processes. IFS=- read -a words <<< $MY_TEXT for word in "${words[@]}"; do MY_INITIALS+=${word:0:1}; done ...

Capitalize all files in a directory using Bash

osx,bash,rename

In Bash 4 you can use parameter expansion directly to capitalize every letter in a word (^^) or just the first letter (^). for f in *; do mv -- "$f" "${f^}" done You can use patterns to form more sophisticated case modifications. But for your specific question, aren't you...

Ignore first few lines and last few lines in a file Linux

linux,awk

awk cannot look ahead so you'll have to save the lines. awk 'NR>2{if(z!="")print z;z=y;y=x;x=$0}' file Practically zero memory overhead...

BASH - conditional sum of columns and rows in csv file

linux,bash,csv,awk

This awk program will print the modified header and modify the output to contain the sums and their division: awk 'BEGIN {FS=OFS=";"} (NR==1) {$10="results/time"; print $0} (NR>1 && NF) {sum8[$10]+=$8; sum9[$10]+=$9; other[$10]=$0} END {for (i in sum8) {$0=other[i]; $8=sum8[i]; $9=sum9[i]; $10=(sum9[i]?sum8[i]/sum9[i]:"NaN"); print}}' which gives: Date;dbms;type;description;W;D;S;results;time;results/time Mon Jun 15 14:22:20 CEST...

Why does `sort file > file` result in an empty file? [duplicate]

bash

With sort afile > afile this happens: The shell opens and truncates afile because of the file direction operation > afile The shell executes the sort program with one argument, afile, and binds stdout of the new process to the file descriptor opened in step 1. The sort program opens...

Aggregate failure codes in bash

bash

Logical or can be done as: main_result=$((main_result || result)) ...

print filenames into scripts in bash

bash,printf,echo

You need to keep format of printf on same line and keep $i outside single quotes: for i in filenames*; do printf '#!/bin/bash #$ -cwd #$ -j y #$ -S /bin/bash #$ -pe threaded 8 $HOME/bin/program -vv -c $HOME/program.key -d '"$i"' --max 10\n' done ...

shell script cut from variables

bash,shell,shellcode

With GNU grep: grep -oP 'aaa&\K.*' file Output: 123 456 \K: ignore everything before pattern matching and ignore pattern itself From man grep: -o, --only-matching Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line. -P, --perl-regexp Interpret PATTERN as a...

Bash modify CSV to change a field

linux,bash,awk

Please save following awk script as awk.src: function date_str(val) { Y = substr(val,0,4); M = substr(val,5,2); D = substr(val,7,2); date = sprintf("%s-%s-%s",Y,M,D); return date; } function time_str(val) { h = substr(val,9,2); m = substr(val,11,2); s = substr(val,13,2); time = sprintf("%s:%s:%s",h,m,s); return time; } BEGIN { FS="|" } # ## MAIN...

Macports switch PHP CLI version

php,bash,drupal,terminal,macports

Do not modify files in /usr/bin. That's Apple's turf, and there are always other possibilities to avoid changing things there, especially since Apple's next update will happily revert these changes again and scripts might rely on /usr/bin/php being exactly the version Apple shipped with the OS. Put the original binary...

Replace [a-z],[a-z] with [a-z], [a-z] and keep the letters

bash,awk,sed

What I have tried sed 's/[a-z],[a-z]/[a-z], [a-z]/g' <<< "suny stony brook, stony brook,usa." You need to use regex's capture groups here to refer to the original [a-z] values. For example: s/$[a-z]$,$[a-z]$/\1, \2/g Notice how I've surrounded those [a-z] with $ and $? These form capture groups that can be...

How to change svn:externals from bash file non-interactive

bash,svn,svn-externals

Use svn ps svn:externals svn://hostname/branchname -F extenals.txt http://svnbook.red-bean.com/en/1.8/svn.ref.svn.c.propset.html...

Identifying when a file is changed- Bash

bash,shell,unix

I would store the output of find, and if non-empty, echo the line break: found=$(find . -name "${myarray[i]}") if [[ -n $found ]]; then { echo "$found"; echo "<br>"; } >> "$tmp" fi ...

Bash alias function with predefined argument

bash

Simply, add it in your .bashrc like: alias gl10="gitLog 10" To apply the changes: source ~/.bashrc...

Why can I view some Unix executable files in Mac OS X and not others?

git,bash,shell,unix,binary

Executable files may be scripts (in which case you can read the text), or binaries (which are ELF formatted machine code). Your shell script is a script; git is an ELF binary. You can use the file command to see more detail. For example, on my nearest Linux system: $...

Permission denied in find: why do we need 2>&1?

bash,error-handling,find,io-redirection

Because the 'permission denied' message is printed in stderr not stdout. 1 is stdout 2 is stderr & specifies that whatever following is a file descriptor not filename 2>&1 redirects stderr to stdout and enables the error message to be piped into the grep command. If excluding permission denied message...

how to modify an array value with given index?

arrays,linux,bash

You don't need the quotes. Just use ${i}, or even $i: pomme[${i}]="" Or pomme[$i]="" ...

How to test if a command is a shell reserved word?

bash,shell

#!/bin/bash string=$1 if [[ $(type "$string" 2>&1) == "$string is a shell"* ]]; then echo "Keyword $string is reserved by shell" fi ...

Extra backslash when storing grep in a value

linux,bash

The output from set -x uses single quotes. So the outer double quotes were replaced with single quotes but you can't escape single quotes inside a single quoted string so when it then replaced the inner double quotes it needed, instead, to replace them with '\'' which ends the single...

how to deletes line from a text file that are taken from another file [duplicate]

shell,awk,sed,grep,sh

Something like this with grep: grep -vxf lines.txt data.txt > no_dupplicate_lines.txt Sample: AMD$ cat lines.txt Line2 Line4 AMD$ cat data.txt Line1 Line2 Line3 Line4 Line5 AMD$ grep -vxf lines.txt data.txt Line1 Line3 Line5 Print the lines that are not matching (-v) the exact lines (-x) from the file lines.txt (-f...

bash interactive script pass input

bash,command-line-arguments

If you want to redirect the normal standard input of the program, you could use so called "here documents" (see e.g. the BASH manual page): java -jar script.jar <<EOF your input here EOF That means standard input (a.k.a. stdin) is redirected and will be the text in the "here document",...

shell script for counting replacements

bash,replace,count

Assuming you want to replace the word 'apple' with 'banana' (exact match) in the contents of the files and not on the names of the files (see my comment above) and that you are using the bash shell: #!/bin/bash COUNTER=0 for file in *.txt ; do COUNTER=$(grep -o "\<apple\>" $file...

AWK write to new column base on if else of other column

linux,bash,shell,awk,sed

You can use: awk -F, 'NR>1 {$0 = $0 FS (($4 >= 0.7) ? 1 : 0)} 1' test_file.csv ...

How do I check whether a file or file directory exist in bash?

bash,if-statement

Checking file and/or directory existence To check whether a file exists in bash, you use the -f operator. For directories, use -d. Example usage: $ mkdir dir $ [ -d dir ] && echo exists! exists! $ rmdir dir $ [ -d dir ] && echo exists! $ touch file...

Bash script using sed acts differently when passing variable

bash,sed

Your variable is still within single quote hence not getting expanded. Use: sed -i 's|^$'"$CHECK"' = $*.|\1'6'|' /user/file.txt ...

Bash script that removes C source comments

bash

Pass all your files through a sed command like this: sed -i "s#[[:space:]]*//.*##g" filepath If you want to reserve comments which have codes before it (like i++;//comment), then: sed -i "/^[[:space:]]*\/\/.*/d" filepath ...

How do I silence the HEAD of a curl request while using the silent flag?

bash,shell,curl,command-line,pipe

Try this: curl --silent "www.site.com" > file.txt ...

AWK count number of times a term appear with respect to other columns

linux,shell,command-line,awk,sed

Almost same as the other answer, but printing 0 instead of blank. AMD$ awk -F, 'NR>1{a[$2]+=$3;b[$2]++} END{for(i in a)print i, a[i], b[i]}' File pear 1 1 apple 2 3 orange 0 1 peach 0 1 Taking , as field seperator. For all lines except the first, update array a. i.e...