Tag Archives: AWK

sed help adding parenthis

By BeefStu

I have the following data and want to put parenthis around the numbers:

Code:


PARTITION PERIOD_MIN VALUES LESS THAN 10649 TABLESPACE ODS_DAILY_MF_AUM,
PARTITION PERIOD_10649 VALUES LESS THAN 10650 TABLESPACE ODS_DAILY_MF_AUM,
PARTITION PERIOD_10650 VALUES LESS THAN 10651 TABLESPACE ODS_DAILY_MF_AUM,
...
...
...


The output should look like this

Code:


PARTITION PERIOD_MIN VALUES LESS THAN (10649) TABLESPACE ODS_DAILY_MF_AUM,
PARTITION PERIOD_10649 VALUES LESS THAN (10650) TABLESPACE ODS_DAILY_MF_AUM,
PARTITION PERIOD_10650 VALUES LESS THAN 1(0651)TABLESPACE ODS_DAILY_MF_AUM,
...
...
...


Can somebody provide me with a SED or AWK statement that can do this in one shot.

Thanks to all who answer

…read more

Source: FULL ARTICLE at The UNIX and Linux Forums

Script to create unique look-up for headers for a Dictionary

By gimley

I have a text file in UTF-8 format which has the following data structure

Code:

HEADWORD=gloss1,gloss2,gloss3 etc


I want to convert it so that all the glosses of the HeadWord appear on separate lines

Code:

HEADWORD=gloss1
HEADWORD=gloss2
HEADWORD=gloss3


An example will illustrate the requirement
INPUT

Code:

હોશમાં આવવું=regain consciousness.
હોશિયાર=clever, intelligent; skilful; alert, vigilant; cautious; understanding, sensible.
હોશિયારી કરવી=boast,(try to) be clever.
હોશિયારી દાખવવી=boast,(try to) be clever.
હોશિયારી બતાવવી=boast,(try to) be clever.
હોશિયારી મારવી=boast,(try to) be clever.
હોશિયારી રાખવી=be cautious,be vigilant,be alert.
હોશિયારી=cleverness, vigilance
હોહા=noise, uproar, tumult, public talk or discussion, excitement, agitation, alarm, consternation.
હોહાકાર=uproar, tumult, excitement, alarm.
હોહો=noise, uproar, tumult, public talk or discussion, excitement, agitation, alarm, consternation.


The Output would be

Code:

હોશિયાર=clever
હોશિયાર=intelligent
હોશિયાર=skilful
હોશિયાર=alert
હોશિયાર=vigilant
હોશિયાર=cautious
હોશિયાર=understanding
હોશિયાર=sensible.
હોશિયારી કરવી=boast
હોશિયારી કરવી=(try to) be clever.
હોશિયારી દાખવવી=boast
હોશિયારી દાખવવી=(try to) be clever.
હોશિયારી બતાવવી=boast
હોશિયારી બતાવવી=(try to) be clever.
હોશિયારી મારવી=boast
હોશિયારી મારવી=(try to) be clever.
હોશિયારી રાખવી=be cautious
હોશિયારી રાખવી=vigilant or alert.
હોશિયારી=cleverness
હોશિયારી=vigilance
હોશિયારી=etc.
હોહા=noise
હોહા=uproar
હોહા=tumult
હોહા=public talk or discussion
હોહા=excitement
હોહા=agitation
હોહા=alarm
હોહા=consternation.
હોહાકાર=uproar
હોહાકાર=tumult
હોહાકાર=excitement
હોહાકાર=alarm
હોહો=noise
હોહો=uproar
હોહો=tumult
હોહો=public talk or discussion
હોહો=excitement
હોહો=agitation
હોહો=alarm
હોહો=consternation


At present I use macros which identify the delimiter, copy the text between two delimiters, paste it on next line, preface it with the headword and continue the operation till end of line and repeat the same for the next line. Since the file is huge a PERL or AWK script would help.
I work under Windows and UNIX type solutions do not work for me unfortunately.
Many thanks in advance.

From: http://www.unix.com/shell-programming-scripting/221547-script-create-unique-look-up-headers-dictionary.html

Help with awk

By karumudi7

Hi,

I have a stream of data like below:

Code:

abcdef1CXYZ1999PQR
ghijkl2MNOJ2012GHI


By using AWK and substr I am extracting the data, but I need to perform a substr based on the year. Say if Year is less than 2011 then I have to extract 9th char for output 3rd column and if Year is greater than or equal 2011, I have to extract 11th char for output 3rd column.

Code:

abc|def|C|1999
ghi|jkl|J|2012


Thanks.

…read more

Source: FULL ARTICLE at The UNIX and Linux Forums

awk and a variable

By andresgom

Hi everybody.


I’m using the AWK command in a simple script.
I’m using this command like this:



$ cat /etc/passwd | awk -F: ‘$1==”proxy” {print}’
Its useful, I have this output:

proxy:x:13:13:proxy:/bin:/bin/sh


But now I need to use a variable instead the string “proxy”, but now I’m trying something like this, but it doesnt work:

THEVARIABLE=proxy
cat /etc/passwd | awk -F: ‘$1==$THEVARIABLE {print}’


The output it’s very different, its like the comparison “$1==”proxy” doesn’t work.



Can anybody hel me? how can I use a Variable into a awk or grep expression?


thanks!!


Andres

…read more

Source: FULL ARTICLE at The UNIX and Linux Forums

Help summing a file using awk

By Drenhead

I’m trying to sum a text file using AWK. Here is an example of the file:

Code:

600|3H68| 46
600|3H69| 46
600|3H6F| 290
600|3H6G| 24
600|3HDY| 1
600|3HDY| 3
600|3HE0| 1
600|3HE0| 3


I would like to sum the third field if the first two fields are the same.

For example, the last 2 lines, I would like them to sum up and have
600|3HE0| 4

Is this possible using AWK?

I tried something like this, but it gave strange results:

Code:

awk 'BEGIN { FS = "|" } ; '{ arr[$1 "|" $2] += $3 } END {for (i in arr) {print i "|" arr[i] } }' count_all.txt


I appreciate any help you can provide.

…read more
Source: FULL ARTICLE at The UNIX and Linux Forums

Forking in Cygwin is too forking slow! Options for BASH in a windows environment?

By Michael Stora

Unfortunately I’m working on scripts that have to be deployed in a Windows environment.

I’m writing some big scripts to scrape data from machine log file into a databases and post-process them. As long as I use BASH built-ins and AWK internal commands the scripts scream. When I have to manipulate the data with conditionals to join in other data and do calculations (including one or more call to bc for floating point calculations on every line), use a lot of pipes, etc., things crawl. Apparently this is due to Cygwin being very thin and efficient for just about everything except fork() emulation which lacks a windows API equivalent.

I could probably do more within AWK but my AWK programming is pretty weak.

I’m getting modest improvements by launching Cygwin with a high priority through a .bat file.

Code:

C:
chdir C:cygwinbin
set CYGWIN=nodosfilewarning
start /B /high /wait bash --login -i ~/bvd.sh
set /p dummyUserInput= Hit return to exit.


What are my options for either speeding up forking or a faster BASH command line environment of Windows?

Mike

…read more
Source: FULL ARTICLE at The UNIX and Linux Forums

Run shell script alternate week per month

By krupasindhu18

Hi All,

Requirement :-
I have shell script in kern shell ,have to run alternate week per month
example-today’s date is 27 Mar 2013 and my script will run today then for the next time when it will run it should check 1st what was the last run date(27 Mar 2013) if it is matched 15days from current date then run else sent a mail with waning mgs.

i have a AWK command it is checking only within a month
date +%Y:%m:%d|awk -vFS=”:” -vOFS=”:” ‘{$3=$3-3;print}’
suppose i am checking 20days from current date it is give some negative value
ex-2013:03:-8

Please give some advice me so i can move forward.

Thank you All.

Regards,
Krupa

…read more
Source: FULL ARTICLE at The UNIX and Linux Forums

Sorting a file with frequency on length

By gimley

Hello,
I have a file which has the following structure

Code:

word space Frequency


The file is around 30,000 headwords each along with its frequency. The words have different lengths. What I need is a PERL or AWK script which can sort the file on length of the headword and once the file is sorted on lenght: smallest to largest; sort each such set of words having the same length on their frequency.
At present I do this in Excel using the

Code:

=Len(text)


formula, but this is getting tedious.
I am giving below a sample input file

Code:

the 29962169
and 14291859
you 12345509
for 3296048
not 3091071
but 2994482
say 2345958
she 2123744
get 2081392
one 1988291
can 1915289
out 1812292
him 1571291
who 1543711
are 1487971
now 1453264
was 1399013
that 7834407
have 5930242
with 3983564
this 3814998
what 3327049
they 2684414
your 2329896
know 2221467
from 2207336
like 1845600
just 1756270
here 1558771
come 1541623
when 1465219
there 1957160
about 1903238
right 1410555
think 1398723
would 1346905


The expected output would be:

Code:

the 29962169
and 14291859
you 12345509
for 3296048
not 3091071
but 2994482
say 2345958
she 2123744
get 2081392
one 1988291
can 1915289
out 1812292
him 1571291
who 1543711
are 1487971
now 1453264
was 1399013
that 7834407
have 5930242
with 3983564
this 3814998
what 3327049
they 2684414
your 2329896
know 2221467
from 2207336
like 1845600
just 1756270
here 1558771
come 1541623
when 1465219
there 1957160
about 1903238
right 1410555
think 1398723
would 1346905


As you can see the file has been sorted on length and then on frequency.
Any help given would avoid the tedium of loading the file each time in excel. Many thanks in advance

…read more
Source: FULL ARTICLE at The UNIX and Linux Forums

Manipulating xml data with awk

By hayreter

Hi everyone,

I have a little bit of complicated task to finish with AWK. Here it is;
I have a data file in xml format which looks like this

Code:


a1 a2 a3 a4 a5
b1 b2 b3 b4 b5
c1 c2 c3 c4 c5
d1 d2 d3 d4 d5
e1 e2 e3 e4 e5


lets say each data block contains 5 rows and 5 columns, what I need to do is this;
I have a condition, and I need to find the row that satisfies this condition then
I need to add an extra field to each row whose value will be calculated using
the columns of the row that satisfies the condition as well as other columns in
other rows. As an example, lets say row “c” satisfies my condition, then I add
an extra field to data which will look like this

Code:


a1 a2 a3 a4 a5 a6
b1 b2 b3 b4 b5 b6
c1 c2 c3 c4 c5 c6
d1 d2 d3 d4 d5 d6
e1 e2 e3 e4 e5 e6


where the last fields are calculated as following;

Code:

a6 = c2*a2 + c3*a3 + c4*a4 + c5*a5
b6 = c2*b2 + c3*b3 + c4*b4 + c5*b5
c6 = c2*c2 + c3*c3 + c4*c4 + c5*c5
d6 = c2*d2 + c3*d3 + c4*d4 + c5*d5
e6 = c2*e2 + c3*e3 + c4*e4 + c5*e5


the algebra on the above calculation may not necessarily be simple as this.

Thanks for any help.

…read more
Source: FULL ARTICLE at The UNIX and Linux Forums

Ryan Kavanagh: Search RCS and CVS ",v" files with rcsgrep

A few years ago I was doing research comparing how large software distributions handled shared object libraries, and studied Debian, FreeBSD, and Ubuntu. Extracting data about Debian packages was easy thanks to Peter Palfrader’s snapshot.debian.org service, which provides a machine-usable interface to Debian’s package history. FreeBSD’s data is equally accessible, albeit in a less pleasant format: their ports tree was stored in CVS until July 2012. One could easily rsync a copy of the ports tree’s CVS repository to a local machine to analyze the data. This left you with a local tree full of ,v files, each corresponding to the history of a given file with at that location. I needed to extract all kinds of data from a tree full of these files, such as what revisions contained lines matching a regex, when these revisions were checked in, any tags associated with it, etc. To make things easier, it also helped to know the line numbers of the matching lines. Hence the birth of rcsgrep.

rcsgrep is a Python script that makes use of Paul McGuire’s fabulous pyparsing library. It allows you to search a RCS file (the ,v file format used by RCS and CVS to store revision history) using a Python regex, and the output format is customizable to allow printing only certain kinds of information, such as the revision number, the line number, the matching line, the line’s author, the date it appeared, any tags associated with the line, and (useful when running over a large number of files) the file name. To make machine parsing (using AWK of course) easier, you can also specify the column separator.

For example, I entered the lines “The quick brown”, “fox jumped over”, “the lazy dog. Woof!” into the file abc, checking in the changes after each line. The invocation ./rcsgrep -s ' ' -f rlLda '.*' abc,v, with spaces for column separation, and format options r is for revision, l for line number, L for line contents, d for date, and a for author, outputs:

1.3    1    The quick brown    2013.02.20.14.24.09    ryan
1.3    2    jumped over the    2013.02.20.14.24.09    ryan
1.3    3    lazy dog. Woof!    2013.02.20.14.24.09    ryan
1.2    1    The quick brown    2013.02.20.14.23.48    ryan
1.2    2    jumped over the    2013.02.20.14.23.48    ryan
1.1    1    The quick brown    2013.02.20.14.23.25    ryan

I’m particularly proud about my grep() function in rcsfile.py, which goes through each revision, tracking additions and deletions, but only keeping the lines matching the regex in memory. In any case, rcsgrep is licensed under the ISC license and can be found on github.

Addendum: I learned after the fact that O’Reilly’s <a target=_blank …read more
Source: FULL ARTICLE at Planet Ubuntu

awk Help – Beginner

By mcclunyboy

Hi,

I think I need to use AWK – however I have no experience of it. Can someone help please?

I have a file like this but with many more records – it is fixed width
THIS15021X 799999 XX 00000099999 00008888888 XX 15022013
THISQ15021X 999999 XX 00000099999 00008888888 XX 15022013
BLAS15021X 9999999XX 00000199999 00008888888 XX 15022013

I want the file to look like this again it is fixed width
THIS15021X 0799999 XX 00000099999 00008888888 XX 15022013
THISQ15021X 0999999 XX 00000099999 00008888888 XX 15022013
BLAS15021X 9999999 XX 00000199999 00008888888 XX 15022013

I want to add a leading 0 to the 2nd coloumn if it contains the word this at the beginning

…read more
Source: FULL ARTICLE at The UNIX and Linux Forums

Complex awk problem

By dietmar13

hello,

i have a complex awk problem…

i have two tables, one with a value (0 to 1) and it’s corresponding p-value, like this:

1. table:
______________________________
value p-value
… …
0.254 0.003
0.245 0.005
0.233 0.006
… …
______________________________

and a second with millions of values (0 to 1), like this:
2. table
______________________________

0.252…
0.234…
0.256…

______________________________

now I have to map the second list to the first table so that I get for each value the corresponding p-value (i.e. the p-value corresponding to the LOWER value in the 1. table).

expexted output:
______________________________
… …
0.252… 0.005
0.234… 0.006
9.256… 0.003
… …
______________________________

one possibility would be to create an indexed array with 1000 instances in this way:

a[0.233]=0.006
a[0.234]=0.006

a[0.245]=0.005

then create a substring of the value of the second list like x.xxx and use this as index for the array.

is there an easier way?

thank you,

dietmar

PS: I have to use AWK, because I already read 7 several gbyte large files an combine these in one table (in minutes) and I have to convert one column on the fly to the corresponding p-values. I think every other programming language will be slower (perhaps except C++, which I can’t program).

…read more
Source: FULL ARTICLE at The UNIX and Linux Forums

Matching two files with awk

By Homa

Hello,

I have two files as follow:
[CODE]AFFY_ID RS_ID CHROMOSOME POS_START POS_END ALLELE1 ALLELE2
SNP_A-1780283 rs17011450 chr4 127630275 127630276 C T
SNP_A-1780285 rs6919430 chr6 90919464 90919465 A C
SNP_A-1780286 — chr7 104281409 104281410 A G
SNP_A-1780301 rs2342723 chr16 5748790 5748791 C T[/CODE]

And the second file:


SNP_A-1780301 2 181.55 G A
SNP_A-1780305 2 140.448 C T
SNP_A-1780321 2 142.446 T C
SNP_A-1780346 2 6.62269 T C
SNP_A-1780368 2 137.474 A G


What I need is to match the first columns of two files and print the second column of the first file into the second file, so, the output would be:

SNP_A-1780301 rs2342723 2 181.55 G A

How could I do this with AWK?

Thanks a lot!

Source: FULL ARTICLE at The UNIX and Linux Forums

Get the nth word of mth line in a file

By freezelty

Hi..

May be a simple question but I just began to write unix scripts a week ago, for sorting some huge amount of experiment data, so I got no common sense about unix scripting and really need your helps…

The situation is, I want to read the nth word of mth line in a file, and then store it in a variable. Been searching through the internet and I got an AWK command like this

Code:

var=$(awk "NR==$m{print $1}" input_file)


With this command line above I want to get the first word in line ‘$m’, but it returns the whole line into the $var (words in the file are separated by tab)

In addition, I would need a command to get the ‘$n’th word in the ‘$m’th line, but I guess the command below would not work because the wrong formating of ‘$’

Code:

var=$(awk "NR==$m{print $n}" input_file)


Thank you…

Source: FULL ARTICLE at The UNIX and Linux Forums

Command to extract all columns except the last few from a txt file

By madrazzii

hello,

i have publicly available txt file with little less than 300000 rows. i want to extract from column 1 to column 218 and save it in another text file. i use the cut command but the file is saved with multiple rows from the source file onto a single row in the destination. basically it is not formatted like the source file. the source file is in the below link. i have removed the www since this is my first post and the file name is INP_psf1301.txt

cms.gov/Medicare/Medicare-Fee-for-Service-Payment/ProspMedicareFeeSvcPmtGen/Downloads/INP_psf1301.zip

i use the following


cut -c1, 218 INP_psf1301.txt > outputfile.txt


but the resulting file is is not formatted like the original. i want the resulting file to be exactly the same except without the last few columns. i want columns 1 to 218 in the resulting file.

is there a line break or something that i should mention or would AWK command work better?

thanks

Source: FULL ARTICLE at The UNIX and Linux Forums

Manipulating sed Direct Input to Direct Output

By the0nion

Hi guys,
been scratching round the forums and my mountain of resources.
Maybe I havn’t read deep enough

My question is not how sed edits a stream and outputs it to a file, rather something like this below:

I have a .txt with some text in it :rolleyes:

abc:123:xyz
123:abc:987
qwe:145:123

Is there a way, to use sed to get, say “123:abc:987”, convert it to string “123,abc,$987” and standard output it to screen? No file overwrite allowed

I have tried things like
1)
sed -i “s/:/,/g” file.txt –> this of course will change all “:” to “,”
2)
sed -n /123,abc,/p file.txt –> print only matching
3)
sed -i “s/,/:/g” file.txt –> convert back to “:”

I have tried some piping but i think I may have the wrong idea of pipe (eg. “| echo” and wondering why no output)

Could you guys throw me some ideas?
Some of my other friends was mentioning AWK.
Note that stupid “$” 😡

Source: FULL ARTICLE at The UNIX and Linux Forums

Sorting on length with identification of number of characters

By gimley

Hello,
I am writing an open-source stemmer in Java for Indic languages which admit a large number of suffixes.
The Java stemmer requires that each suffix string be sorted as per its length and that all strings of the same length are arranged in a single group, sorted alphabetically. Moreover as a header I need to specify the numeric value of the string, say

Quote:

5
6
7
8
etc.


Since the languages in question have over 300 and more suffixes, trying to sort on length and identifying the length of each string and counting it becomes a difficult issue.
An example will make this clear.
Input:

Quote:

आधी
इतक
इतपत
ईचना
ईचनात

ईना
ईन


Expected output

Quote:

1

2
ईन
3
आधी
इतक
ईना
4
इतपत
ईचना
5
ईचनात


Since handling such a large database is laborious, is it possible to write a script in AWK or PERL which would enable the above output.
Your help would go a long way in putting java-based stemmers in different languages in the open-source community.
Many thanks in advance for your kind help

Source: FULL ARTICLE at The UNIX and Linux Forums

Help in separating a multilingual file

By gimley

Hello,
I have a text file running into around 100 thousand+ lines which has the following rigid structure:

Quote:

Identity Number (always a Numeric Field)


Quote:

English (a whole set of names


Quote:

Data in another language in UTF8 format


Each field is separated by a comma.
Some examples are given below:

Quote:

23,Chinttaman Pagare,चिंतमण पगारे
24, Chinttaman Pateel,चिंतामण पाटल
25, Chinttaman Rout,चिंतामण राऊत
26, Chinttaman Yashawante,चिंतामण यशवंत


I would like to extract the data such that all the English words are stored in one file and the other language words in another file. The numbers would be ignored.
I work under windows OS.
A script in AWK or PERL would be of great help.
Many thanks in advance

Source: FULL ARTICLE at The UNIX and Linux Forums