loop add a comma after nth comma using awk

I feel like this should be a fairly straight forward question, but I cant seem to get it to work.

I have a csv file and I need to add comma after the nth comma in each row. I believe I have to use gsub to get it to loop. something like

{gsub(/$nth/,/",")}

but I don't understand awk well enough to get it to work.

The end goal of my script is to check to see if a word exist and if it does NOT add a comma after the nth comma.

I'm using grep for that part like this:

TEST1=$(cat $file | grep 'Sentence Skills')
 if [ $? -eq 1 ]
  then
  awk command to add comma after nth comma
 fi

If it doesnt exist I need to add a comma after the nth comma to make sure everything lines up correctly

UPDATE to clarify with sample input and output (my apologies, I originally was not going to include the grep if then fi part)

Here is a sample of the csv:

last,first,A00XXXXXX,1888-01-01,2015-05-13,Reading Comprehension 97,Sentence Skills 104,College Level Math 76,Elementary Algebra 115,
last,first,A00XXXXXX,1888-01-01,2015-05-13,Elementary Algebra 34,
last,first,A00XXXXXX,1888-01-01,2015-05-13,College Level Math 64,Elementary Algebra 114,
last,first,A00XXXXXX,1888-01-01,2015-05-13,Reading Comprehension 87,College Level Math 64,Elementary Algebra 114,

And this is what I need it to look like:

last,first,A00XXXXXX,1888-01-01,2015-05-13,Reading Comprehension 97,Sentence Skills 104,College Level Math 76,Elementary Algebra 115,
last,first,A00XXXXXX,1888-01-01,2015-05-13,,,,Elementary Algebra 34,
last,first,A00XXXXXX,1888-01-01,2015-05-13,,,College Level Math 64,Elementary Algebra 114,
last,first,A00XXXXXX,1888-01-01,2015-05-13,Reading Comprehension 87,,College Level Math 64,Elementary Algebra 114,

I need to add one comma after the 5th comma if Reading comprehension doesn't exist, then add one comma after the 6th comma if Sentence Skills doesn't exist, then one comma after the 7th comma if College Level Math doesn't exist, then one comma if Elementary algebra doesn't exist.

If any of those do exist, it does not add a comma and skips on to the next one.

Best How To :

In my opinion it can be a difficult task to solve it with awk. Here you have an approach with perl. The difference is that I can use a hash, sort it and add items in the middle of an array with splice.

perl -F',' -lanE '
    BEGIN {
        %h = ( 
            5 => q|Reading Comprehension|,
            6 => q|Sentence Skills|,
            7 => q|College Level Math|,
            8 => q|Elementary Algebra|,
        );  
    };  
    for (sort keys %h) {
        if ($F[$_] !~ m/^$h{$_}/) {
            splice @F, $_, 0, q||;
            ++$_;
        }   
    }
    printf qq|%s,\n|, join q|,|, @F;
' infile

The -F has same meaning that awk, so I loop over the hash and check if that field number contains the value. If it doesn't match, insert an empty element just before it, and the last printf joins all fields with comma. It yields:

last,first,A00XXXXXX,1888-01-01,2015-05-13,Reading Comprehension 97,Sentence Skills 104,College Level Math 76,Elementary Algebra 115,
last,first,A00XXXXXX,1888-01-01,2015-05-13,,,,Elementary Algebra 34,
last,first,A00XXXXXX,1888-01-01,2015-05-13,,,College Level Math 64,Elementary Algebra 114,
last,first,A00XXXXXX,1888-01-01,2015-05-13,Reading Comprehension 87,,College Level Math 64,Elementary Algebra 114,

Finding the average of a column excluding certain rows using AWK

linux,bash,awk,scripting

Through awk, $ awk '$5!="99999"{sum+=$5}END{print sum}' file 227.5 Explanation: $5!="99999" if 5th column does not contain 99999, then do {sum+=$5} adding the value of 5th column to the variable sum. Likewise it keeps adding the value of 5th column when awk see's the record which satisfies the given condition. Finally...

Replace [a-z],[a-z] with [a-z], [a-z] and keep the letters

bash,awk,sed

What I have tried sed 's/[a-z],[a-z]/[a-z], [a-z]/g' <<< "suny stony brook, stony brook,usa." You need to use regex's capture groups here to refer to the original [a-z] values. For example: s/$[a-z]$,$[a-z]$/\1, \2/g Notice how I've surrounded those [a-z] with $ and $? These form capture groups that can be...

Macports switch PHP CLI version

php,bash,drupal,terminal,macports

Do not modify files in /usr/bin. That's Apple's turf, and there are always other possibilities to avoid changing things there, especially since Apple's next update will happily revert these changes again and scripts might rely on /usr/bin/php being exactly the version Apple shipped with the OS. Put the original binary...

Bash script that removes C source comments

bash

Pass all your files through a sed command like this: sed -i "s#[[:space:]]*//.*##g" filepath If you want to reserve comments which have codes before it (like i++;//comment), then: sed -i "/^[[:space:]]*\/\/.*/d" filepath ...

How do I silence the HEAD of a curl request while using the silent flag?

bash,shell,curl,command-line,pipe

Try this: curl --silent "www.site.com" > file.txt ...

BASH - conditional sum of columns and rows in csv file

linux,bash,csv,awk

This awk program will print the modified header and modify the output to contain the sums and their division: awk 'BEGIN {FS=OFS=";"} (NR==1) {$10="results/time"; print $0} (NR>1 && NF) {sum8[$10]+=$8; sum9[$10]+=$9; other[$10]=$0} END {for (i in sum8) {$0=other[i]; $8=sum8[i]; $9=sum9[i]; $10=(sum9[i]?sum8[i]/sum9[i]:"NaN"); print}}' which gives: Date;dbms;type;description;W;D;S;results;time;results/time Mon Jun 15 14:22:20 CEST...

Permission denied in find: why do we need 2>&1?

bash,error-handling,find,io-redirection

Because the 'permission denied' message is printed in stderr not stdout. 1 is stdout 2 is stderr & specifies that whatever following is a file descriptor not filename 2>&1 redirects stderr to stdout and enables the error message to be piped into the grep command. If excluding permission denied message...

AWK write to new column base on if else of other column

linux,bash,shell,awk,sed

You can use: awk -F, 'NR>1 {$0 = $0 FS (($4 >= 0.7) ? 1 : 0)} 1' test_file.csv ...

Identifying when a file is changed- Bash

bash,shell,unix

I would store the output of find, and if non-empty, echo the line break: found=$(find . -name "${myarray[i]}") if [[ -n $found ]]; then { echo "$found"; echo "<br>"; } >> "$tmp" fi ...

Ignore first few lines and last few lines in a file Linux

linux,awk

awk cannot look ahead so you'll have to save the lines. awk 'NR>2{if(z!="")print z;z=y;y=x;x=$0}' file Practically zero memory overhead...

how to immediately login through ssh?

bash,ssh

SSH logins can be passwordless with the use of key authentication. Arch has a great documentation page on how to set it up, with steps that should work on most ditributions: https://wiki.archlinux.org/index.php/SSH_keys It boils down to these basic steps (details in linked document): Generate keypair Copy public key to authorized...

Aggregate failure codes in bash

bash

Logical or can be done as: main_result=$((main_result || result)) ...

How to append entry the end of a multi-line entry using any of stream editors like sed or awk

linux,bash,awk,sed,sh

Here's a sed version: /^Host_Alias/{ # whenever we match Host_Alias at line start : /\\$/{N;b} # if backslash, append next line and repeat s/$/,host25/ # add the new host to end of line } If you need to add your new host to just one of the host aliases, adjust...

How do I check whether a file or file directory exist in bash?

bash,if-statement

Checking file and/or directory existence To check whether a file exists in bash, you use the -f operator. For directories, use -d. Example usage: $ mkdir dir $ [ -d dir ] && echo exists! exists! $ rmdir dir $ [ -d dir ] && echo exists! $ touch file...

How to test if a command is a shell reserved word?

bash,shell

#!/bin/bash string=$1 if [[ $(type "$string" 2>&1) == "$string is a shell"* ]]; then echo "Keyword $string is reserved by shell" fi ...

Extracting columns within a range AWK

unix,awk

You want to test for 0.75-0.8 but wrote code to test for 0.7-0.75 and you forgot to specify what to test in the second part of your condition. Do this: awk '$2 >= 0.75 && $2 <= 0.8' Also note that you want a numeric comparison not a string comparison...

Bash alias function with predefined argument

bash

Simply, add it in your .bashrc like: alias gl10="gitLog 10" To apply the changes: source ~/.bashrc...

While loop in bash using variable from txt file

linux,bash,rhel

As indicated in the comments, you need to provide "something" to your while loop. The while construct is written in a way that will execute with a condition; if a file is given, it will proceed until the read exhausts. #!/bin/bash file=Sheetone.txt while IFS= read -r line do echo sh...

How to extract first letters of dashed separated words in a bash variable?

linux,string,bash,shell,variables

This isn't the shortest method, but it doesn't require any external processes. IFS=- read -a words <<< $MY_TEXT for word in "${words[@]}"; do MY_INITIALS+=${word:0:1}; done ...

How to change svn:externals from bash file non-interactive

bash,svn,svn-externals

Use svn ps svn:externals svn://hostname/branchname -F extenals.txt http://svnbook.red-bean.com/en/1.8/svn.ref.svn.c.propset.html...

Extra backslash when storing grep in a value

linux,bash

The output from set -x uses single quotes. So the outer double quotes were replaced with single quotes but you can't escape single quotes inside a single quoted string so when it then replaced the inner double quotes it needed, instead, to replace them with '\'' which ends the single...

Why can I view some Unix executable files in Mac OS X and not others?

git,bash,shell,unix,binary

Executable files may be scripts (in which case you can read the text), or binaries (which are ELF formatted machine code). Your shell script is a script; git is an ELF binary. You can use the file command to see more detail. For example, on my nearest Linux system: $...

Shell script to loop over files with same names but different extensions

linux,bash,shell

anubhava's solution is excellent if, as they do in your example, the extensions sort into the right order. For the more general case, where sorting cannot be relied upon, we can specify the argument order explicitly: for f in *.ext1 do program "$f" "${f%.ext1}.ext2" done This will work even if...

Python: can't access newly defined environment variables

python,bash,environment-variables

After updating your .bashrc, perform source ~/.bashrc to apply the changes. Also, merge the two BONSAI-related calls into one: export BONSAI=/home/me/Utils/bonsai_v3.2 UPDATE: It was actually an attempt to update the environment for some Eclipse-based IDE. This is a different usecase altogether. It should be described in the Eclipse help. Also,...

AWK count number of times a term appear with respect to other columns

linux,shell,command-line,awk,sed

Almost same as the other answer, but printing 0 instead of blank. AMD$ awk -F, 'NR>1{a[$2]+=$3;b[$2]++} END{for(i in a)print i, a[i], b[i]}' File pear 1 1 apple 2 3 orange 0 1 peach 0 1 Taking , as field seperator. For all lines except the first, update array a. i.e...

linux - running a process and tailing a file simultaneously

bash,shell,tail

I would simply start the tail in background and the python process in foreground. When the python process finishes you can kill the tail, like this: #!/bin/bash touch /tmp/out # Make sure that the file exists tail -f /tmp/out & pid=$! python test.py kill "$pid" ...

how to modify an array value with given index?

arrays,linux,bash

You don't need the quotes. Just use ${i}, or even $i: pomme[${i}]="" Or pomme[$i]="" ...

shell script for counting replacements

bash,replace,count

Assuming you want to replace the word 'apple' with 'banana' (exact match) in the contents of the files and not on the names of the files (see my comment above) and that you are using the bash shell: #!/bin/bash COUNTER=0 for file in *.txt ; do COUNTER=$(grep -o "\<apple\>" $file...

Convert AWK command to sqlite query

sql,awk,sqlite3

SQLite is an embedded database, i.e., it is designed to be used together with a 'real' programming language. It might be possible to import that log file into a database file, but the whole point of having a database is to store the data, which is neither a direct goal...

how to deletes line from a text file that are taken from another file [duplicate]

shell,awk,sed,grep,sh

Something like this with grep: grep -vxf lines.txt data.txt > no_dupplicate_lines.txt Sample: AMD$ cat lines.txt Line2 Line4 AMD$ cat data.txt Line1 Line2 Line3 Line4 Line5 AMD$ grep -vxf lines.txt data.txt Line1 Line3 Line5 Print the lines that are not matching (-v) the exact lines (-x) from the file lines.txt (-f...

awk ternay operator, count fs with ,

awk

awk -F, '{ print ( NF ? $NF : $0 ) }' file ...

Capitalize all files in a directory using Bash

osx,bash,rename

In Bash 4 you can use parameter expansion directly to capitalize every letter in a word (^^) or just the first letter (^). for f in *; do mv -- "$f" "${f^}" done You can use patterns to form more sophisticated case modifications. But for your specific question, aren't you...

using sed to replace a line with back slashes in a shell script

regex,bash,shell,ssh,sed

You can use it with ssh and heredoc like this: ssh -t -t [email protected]<<'EOF' sed 's~out_prefix=orderid ^2\\\\d\\+ updatemtnotif/~out_prefix=orderid ^2\\\\d\\+ updatemtnotif_fr/~' ~/path/to/file exit EOF PS: It is important to quote the 'EOF' as shown....

shell script cut from variables

bash,shell,shellcode

With GNU grep: grep -oP 'aaa&\K.*' file Output: 123 456 \K: ignore everything before pattern matching and ignore pattern itself From man grep: -o, --only-matching Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line. -P, --perl-regexp Interpret PATTERN as a...

Bash script using sed acts differently when passing variable

bash,sed

Your variable is still within single quote hence not getting expanded. Use: sed -i 's|^$'"$CHECK"' = $*.|\1'6'|' /user/file.txt ...

Assign and use of a variable in the same subshell

bash,scope,subshell

Let's look to the POSIX specification to understand why this behaves as it does, not just in bash but in any compliant shell: 2.10.2, Shell Grammar Rules From rule 7(b), covering cases where an assignment precedes a simple command: If all the characters preceding '=' form a valid name (see...

bash interactive script pass input

bash,command-line-arguments

If you want to redirect the normal standard input of the program, you could use so called "here documents" (see e.g. the BASH manual page): java -jar script.jar <<EOF your input here EOF That means standard input (a.k.a. stdin) is redirected and will be the text in the "here document",...

AWK|BASH, use double FS and ternary operator

bash,awk

You can use this awk command awk -F, '{if (split($1, a, " ") > 2) print $NF; else print}' file This will output rosario escuela estadist, medellin medellin If you want to get rid of the space before $NF, use this awk command awk -F, '{if (split($1, a, " ")...

Why does `sort file > file` result in an empty file? [duplicate]

bash

With sort afile > afile this happens: The shell opens and truncates afile because of the file direction operation > afile The shell executes the sort program with one argument, afile, and binds stdout of the new process to the file descriptor opened in step 1. The sort program opens...

print filenames into scripts in bash

bash,printf,echo

You need to keep format of printf on same line and keep $i outside single quotes: for i in filenames*; do printf '#!/bin/bash #$ -cwd #$ -j y #$ -S /bin/bash #$ -pe threaded 8 $HOME/bin/program -vv -c $HOME/program.key -d '"$i"' --max 10\n' done ...

Calling find more than once on the same folder tree

linux,bash,shell,unix,find

Try this: find . -mmin +35 -or -mmin -25 find supports several logical operators (-and, -or, -not). See the OPERATORS section of the man pages for more details. ==================== EDIT: In response to the question about processing the two matches differently, I do not know of a way to do...

Matching string inside file and returning result

regex,string,bash,shell,grep

Using sqlite3 from bash on OS X seems fairly straightforward (I'm no expert at this, by the way). You will need to find out which table you need. You can do this with an interactive session. I'll show you with the database you suggested: /Users/fredbloggs> sqlite3 ~/Library/Application\ Support/Dock/desktoppicture.db SQLite version...

Bash modify CSV to change a field

linux,bash,awk

Please save following awk script as awk.src: function date_str(val) { Y = substr(val,0,4); M = substr(val,5,2); D = substr(val,7,2); date = sprintf("%s-%s-%s",Y,M,D); return date; } function time_str(val) { h = substr(val,9,2); m = substr(val,11,2); s = substr(val,13,2); time = sprintf("%s:%s:%s",h,m,s); return time; } BEGIN { FS="|" } # ## MAIN...