Difference in behaviour between whitespace in IFS vs. non-whitespace in IFS

Tag: shell,posix

I know that read command splits the input by the characters in the IFS. So if I set IFS as a space then input to the read command is split using space as the delimiter. This is shown below.

[[email protected] ~]$ IFS=" "
[[email protected] ~]$ read a b c
 foo bar baz qux
[[email protected] ~]$ echo "$a"
foo
[[email protected] ~]$ echo "$b"
bar
[[email protected] ~]$ echo "$c"
baz qux

I was expecting that if I make change the IFS to a non-whitespace character, say, a colon and use colon as delimiter in my input, there should be no change in behaviour. But this turned out to be wrong. For example, in the output below, echo "$a" is blank. Why?

[[email protected] ~]$ IFS=:
[[email protected] ~]$ read a b c
:foo:bar:baz:qux
[[email protected] ~]$ echo "$a"

[[email protected] ~]$ echo "$b"
foo
[[email protected] ~]$ echo "$c"
bar:baz:qux
[[email protected] ~]# echo $c
bar baz qux

And why does the output of echo $c does not contain the colons?

Best How To :

Question 1: In the output below, echo "$a" is blank. Why?

Because whitespace is special as defined in POSIX (2.6.5 Field Splitting):

3a. IFS white space shall be ignored at the beginning and end of the input.

Question 2: And why does the output of echo $c does not contain the colons?

Because bash performs word splitting in accordance with IFS (which is still set to :) before passing bar, baz and qux as separate arguments to echo.

How to extract first letters of dashed separated words in a bash variable?

linux,string,bash,shell,variables

This isn't the shortest method, but it doesn't require any external processes. IFS=- read -a words <<< $MY_TEXT for word in "${words[@]}"; do MY_INITIALS+=${word:0:1}; done ...

Shell script to loop over files with same names but different extensions

linux,bash,shell

anubhava's solution is excellent if, as they do in your example, the extensions sort into the right order. For the more general case, where sorting cannot be relied upon, we can specify the argument order explicitly: for f in *.ext1 do program "$f" "${f%.ext1}.ext2" done This will work even if...

AWK write to new column base on if else of other column

linux,bash,shell,awk,sed

You can use: awk -F, 'NR>1 {$0 = $0 FS (($4 >= 0.7) ? 1 : 0)} 1' test_file.csv ...

Linux-wget command

linux,shell,wget

Try this to create a string variable n, with no leading whitespace (thanks @011c): n="10.0.0.135.527" wget http://infamvn:8081/nexus/content/groups/LDM_REPO_LIN64/com/infa/com.infa.products.ldm.ingestion.server.scala/"$n"-SNAPSHOT/com.infa.products.ldm.ingestion.server.scala-"$n"-20150622.210643-1-sources.jar ...

adb shell command from batch file

android,shell,adb

Just use the following line in your batch file: adb shell am instrumentation that will connect to the shell on the Android device and run the am command....

Subtract time in r, forcing unit of results to minutes [duplicate]

r,posix,posixct

You can try with difftime df1$time.diff <- with(df1, difftime(time.stamp2, time.stamp1, unit='min')) df1 # time.stamp1 time.stamp2 time.diff #1 2015-01-05 15:00:00 2015-01-05 16:00:00 60 mins #2 2015-01-05 16:00:00 2015-01-05 17:00:00 60 mins #3 2015-01-05 18:00:00 2015-01-05 20:00:00 120 mins #4 2015-01-05 19:00:00 2015-01-05 20:00:00 60 mins #5 2015-01-05 20:00:00 2015-01-05 22:00:00 120...

Need to update a text file with new content found in an other text file

bash,shell,text-files

You could use output redirection cat newtodo.txt > todo.txt or You could rename newtodo.txt mv newtodo.txt todo.txt ...

linux - running a process and tailing a file simultaneously

bash,shell,tail

I would simply start the tail in background and the python process in foreground. When the python process finishes you can kill the tail, like this: #!/bin/bash touch /tmp/out # Make sure that the file exists tail -f /tmp/out & pid=$! python test.py kill "$pid" ...

Reading tab (\t) separated text from file in shell

bash,shell,text-files

In Bash you can do this: #!/bin/bash declare -a varA varB while IFS=$'\t' read -r num first second;do varA+=("$first") varB+=("$second") done <file echo ${varA[1]} ${varB[1]} You can access each element of varA array using index ${varA[$index]} or all of them at once with ${varA[@]}....

Mac OSX - Allow for user input in shell script via GUI or Prompt

osx,bash,shell

From what I understand I would recommend you look in to Applescript as this will allow you to have a GUI Interface as well as executing 'SHELL' commands. First of all I would open 'Script Editor' program that comes preinstalled on Mac's This is an example script which asks for...

how to spy on linux binaries for testing of shell scripts

linux,bash,shell,testing,spy

Just an idea, I didn't see this anywhere but: Unless you're using full paths to invoke those binaries, you could create mocks of those libraries, e.g., in your projects bin/ directory and make that directory be the first in your $PATH. export PATH="$PWD/bin:$PATH" To mock grep, for example, you could...

find numbers divisible by 3 in csv file using shell script

bash,shell,unix,awk

awk -F'|' '{for(i=1;i<=NF;i++)if(!($i%3))print $i}' file this awk one-liner shoud do. With your example, the cmd outputs: 3 6 9 ...

How to test if a command is a shell reserved word?

bash,shell

#!/bin/bash string=$1 if [[ $(type "$string" 2>&1) == "$string is a shell"* ]]; then echo "Keyword $string is reserved by shell" fi ...

Sqoop Export with Missing Data

sql,postgresql,shell,hadoop,sqoop

I solved the problem by changing my reduce function so that if there were not the correct amount of fields to output a certain value and then I was able to use the --input-null-non-string with that value and it worked.

Use Unix Executable File to Run Shell Script and MPKG File

osx,shell,unix

The most common issue when handling variables containing paths of directories and files is the presence of special characters such as spaces. To handle those correctly, you should always quote the variables, using double quotes. Better code would therefor be: sudo sh "$path/join.sh" sudo sh "$path/join2.sh" It is also advised...

Check for decimal point and add it at the end if its not there using awk/perl

regex,perl,shell,awk

In awk Just sub for those fields awk -F, -vOFS="," '{sub(/^[^\.]+$/,"&.",$6);sub(/^[^\.]+$/,"&.",$11)}1' file or sed sed 's/^$\([^,]*,$\{5\}[^.,]\+\),/\1./;s/^$\([^,]*,$\{10\}[^.,]\+\),/\1./' file ...

How to find average and maximum in an interval using Shell [closed]

linux,bash,shell,unix,awk

Please make a search before you ask any question many posts are already there You can try something like below, modify accordingly Input [[email protected] tmp]$ cat input.txt 1 3 2 5 3 4 4 3 5 2 6 1 7 3 8 3 9 4 10 2 11 2 12...

shell script to read each line and run the condition

shell

Are you saying that you want to detect when 5 consecutive rows contain a value greater than 200? If so: awk '{a = $1 > lim ? a + 1 : 0} a > seq {print "alert on line " NR}' lim=200 seq=5 input It's not clear what you actually...

Fail Jenkins job when nosetests fail

python,shell,jenkins,continuous-integration

I suspect nosetests will return a non zero value on failure so you can set the shell to auto fail with set -e or any of the other options in here...

How do I align my output?

shell

I'd rewrite that as: #!/bin/bash while read -ra hello; do name=${hello[8]} if [[ $name == "$1"* ]]; then log=${hello[2]} echo "$log $name" fi done | column -t read -ra splits the input line and stores the words in the "hello" array. [[ $name == "$1"* ]] is a built-in way...

oh-my-zsh error after upgrade: ~/.oh-my-zsh/lib/misc.zsh:3: parse error near `then'

shell,zsh,zshrc,oh-my-zsh

Use upgrade_oh_my_zsh upgrade to the latest version. This issue fix here...

How to extract single-/multiline regex-matching items from an unpredictably formatted file and put each one in a single line into output file?

linux,shell,unix,replace,grep

Assuming that your document is well-formed, i.e. opening tags always match with a closing tag, then this may be what you need: sed '[email protected]<[/]\?b>@\n&\[email protected]' path/to/input.txt | awk 'BEGIN {buf=""} // {Y=1; buf=""} /<\/b>/ {Y=0; print buf""} Y {buf = buf$0} ' | tr -s ' ' Output: data1...

using sed to replace a line with back slashes in a shell script

regex,bash,shell,ssh,sed

You can use it with ssh and heredoc like this: ssh -t -t [email protected]<<'EOF' sed 's~out_prefix=orderid ^2\\\\d\\+ updatemtnotif/~out_prefix=orderid ^2\\\\d\\+ updatemtnotif_fr/~' ~/path/to/file exit EOF PS: It is important to quote the 'EOF' as shown....

Redirect output from file to stdout

bash,shell,unix,stdout

Many tools interpret a - as stdin/stdout depending on the context of its usage. Though, this is not part of the shell and therefore depends on the program used. In your case the following could solve your problem: myprog -o - input_file ...

How do I silence the HEAD of a curl request while using the silent flag?

bash,shell,curl,command-line,pipe

Try this: curl --silent "www.site.com" > file.txt ...

Why can I view some Unix executable files in Mac OS X and not others?

git,bash,shell,unix,binary

Executable files may be scripts (in which case you can read the text), or binaries (which are ELF formatted machine code). Your shell script is a script; git is an ELF binary. You can use the file command to see more detail. For example, on my nearest Linux system: $...

Identifying when a file is changed- Bash

bash,shell,unix

I would store the output of find, and if non-empty, echo the line break: found=$(find . -name "${myarray[i]}") if [[ -n $found ]]; then { echo "$found"; echo " "; } >> "$tmp" fi ...

Delete some lines from text using Linux command

linux,shell,sed,grep,pattern-matching

The -v option to grep inverts the search, reporting only the lines that don't match the pattern. Since you know how to use grep to find the lines to be deleted, using grep -v and the same pattern will give you all the lines to be kept. You can write...

how to deletes line from a text file that are taken from another file [duplicate]

shell,awk,sed,grep,sh

Something like this with grep: grep -vxf lines.txt data.txt > no_dupplicate_lines.txt Sample: AMD$ cat lines.txt Line2 Line4 AMD$ cat data.txt Line1 Line2 Line3 Line4 Line5 AMD$ grep -vxf lines.txt data.txt Line1 Line3 Line5 Print the lines that are not matching (-v) the exact lines (-x) from the file lines.txt (-f...

Matching string inside file and returning result

regex,string,bash,shell,grep

Using sqlite3 from bash on OS X seems fairly straightforward (I'm no expert at this, by the way). You will need to find out which table you need. You can do this with an interactive session. I'll show you with the database you suggested: /Users/fredbloggs> sqlite3 ~/Library/Application\ Support/Dock/desktoppicture.db SQLite version...

Using the shell provided with NodeJS

javascript,node.js,shell,require,node-modules

As per issue #5431, looks like the Node.JS REPL doesn't find globally-installed modules and this is expected behaviour. In the article linked from that issue, it reads: If you’re installing something that you want to use in your program, using require('whatever'), then install it locally, at the root of your...

Storing data in shell

shell,sh

This script will get called and passed a customer number. myEmailFinder "$CustID" I want to populate a variable based on the passed in customer number. emailAddr=$( myEmailFinder "$CustID") I want to pull an e-mail associated with that customer number and populate a variable with it. I would prefer this...

Why does `su` not work in Jenkins?

shell,jenkins,su

su doesn't enter an interactive session when in a non-interactive session the way it does in an interactive session. In a shell script you get to run a single command in the su context su <user> <command>....

shell script cut from variables

bash,shell,shellcode

With GNU grep: grep -oP 'aaa&\K.*' file Output: 123 456 \K: ignore everything before pattern matching and ignore pattern itself From man grep: -o, --only-matching Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line. -P, --perl-regexp Interpret PATTERN as a...

AWK count number of times a term appear with respect to other columns

linux,shell,command-line,awk,sed

Almost same as the other answer, but printing 0 instead of blank. AMD$ awk -F, 'NR>1{a[$2]+=$3;b[$2]++} END{for(i in a)print i, a[i], b[i]}' File pear 1 1 apple 2 3 orange 0 1 peach 0 1 Taking , as field seperator. For all lines except the first, update array a. i.e...

Allowing an executable to be terminated w/ ctrl+c in a shell script w/o exiting

bash,shell

The simplest way to do it is to trap the Ctrl+C signal to do nothing but pass the control to the shell script again. I tried the below code and it worked for me. Replace the sleep commands by the real commands you want to execute. #!/bin/bash #Trap the Ctrl+C...

shell script to shutdown/restart Linux system

linux,shell

Yes, use the -S switch which reads the password from STDIN: $echo <password> | sudo -S <command> Exposing your password is generally bad idea search for something that can protect / hide it. In the past I've used Jenkins plugins to do this while executing the scripts regularly....

Counting bytes received by posix read()

c,function,serial-port,posix

Yes, temp_uart_count will contain the actual number of bytes read, and obviously that number will be smaller or equal to the number of elements of temp_uart_data. If you get 0, it means that the end of file (or an equivalent condition) has been reached and there is nothing else to...

Using AWK to parse fields with commas

regex,bash,shell,awk

I'm not willing to wade through your whole question (sorry, IMHO it's just too long with too much extraneous information) but it looks like you're trying to extract the individual fields from that "confile1" at the top of your question so maybe this is all the hint you need: $...

Calling find more than once on the same folder tree

linux,bash,shell,unix,find

Try this: find . -mmin +35 -or -mmin -25 find supports several logical operators (-and, -or, -not). See the OPERATORS section of the man pages for more details. ==================== EDIT: In response to the question about processing the two matches differently, I do not know of a way to do...

How to replace newlines/linebreaks with a single space, but only if they are inside of start/end regexes?

regex,linux,shell,unix,replace

You can do it using awk as: awk '/\[x/{f=1} {if(f)printf "%s",$0; else print $0;} /y\]/{print ""; f=0}' Output: [x data1 data2 data3 data4 y] [a data5 data 6 data7 data 8 b> [x data y] You can also simplify to: awk '/\[x/,/y\]/{ORS=""; if(/y\]/) ORS="\n";}{print}' Output: [x data1 data2 data3 data4...

docker run local script without host volumes

database,shell,docker,docker-compose

The solutions I have found are: docker run tomdavidson/initdb bash -c "`cat initdb.sh`" and Set an ENV VAR equal to your script and set up your Docker image to run the script (of course one can ADD/COPY and use host volumes but that is not this question), for example: docker...

How to store command line arguments for future use?

bash,shell

Store all the arguments in an array: args=("[email protected]") Then print them as: printf "%s\n" "${args[@]}" ...

Piping echo into sendmail in rc.local fails

linux,shell,email,sendmail,boot

Problem in using bash/zsh vs /bin/sh Bash and Zsh have builtin echo with option -e, but a system /bin/echo - not. Compare: /bin/echo -e "TO:[email protected] \nSUBJECT:System Booted \n\nBeware I live! \n" echo -e "TO:[email protected] \nSUBJECT:System Booted \n\nBeware I live! \n" You may use script like this: #!/bin/sh sendmail -t -vs...

Export database from a remote server into a specific folder on my computer

php,mysql,database,shell,command-line-interface

You can run mysqldump -h yourhostname -u youruser -p yourdatabasename > C:\your\file\path.sql -h connects you to the remote servers IP so you can dump the data locally, as long as your user has the correct privileges and you can connect remotely to your database server. However, you may need to...

Multiple line search in a file using java or unix command

java,shell,unix,command

You can find like this. File file = new File("data/pattern.txt"); Pattern pat = Pattern.compile("subclass \"Pool1\" 11:22:33:44:55:66 \\{\\s*dynamic;\\s*\\}"); String content = Files.lines(file.toPath()).collect(Collectors.joining("\n")); Matcher m = pat.matcher(content); while (m.find()) { System.out.printf("found at %d-%d%n", m.start(), m.end()); } ...

Shell script file, checks if text file is empty or not

shell,text-files

when i run my code i get the following ./todo.sh: line 25: syntax error near unexpected token else' ./todo.sh: line 25: else' You must use parentheses to define a shell function, but they have no part in calling one. A shell function is invoked just like any other command:...

Missing one condition to display a string

shell

The 'grep' command uses a regular expression to match text. Use a '^' before the expression, to match from the start of the line. So, you can change the line name=$(echo $hello | cut -d' ' -f9 | grep $1) to name=$(echo $hello | cut -d' ' -f9 | grep...

Group of data in shell language

linux,bash,shell

In x="1 2 3" for i in $x what you are doing is for i in 1 2 3:. But in second case doing xm=`cat $x` ym=`cat $y` This might lead to error saying No such file or directory is the echo $x and echo $y file does not exit....

Shell script to check if mysql is up or down

mysql,shell,cron

You can use below script #!/bin/bash USER=root PASS=root123 mysqladmin -h remote_server_ip -u$USER -p$PASS processlist ###user should have mysql permission on remote server. Ideally you should use different user than root. if [ $? -eq 0 ] then echo "do nothing" else ssh remote_server_ip ###remote server linux root server password should...