Tag Archives: YYYYMMDD

Linux

Fast processing(mv command) of 1 million+ files using find, mv and xargs

Leave a comment

By agentgrecko

Hi, I’d like to ask if anybody can help improve my code to move 1 million+ files from a directory to another:

Code:
find /source/dir -name file* -type f | xargs -I '{}' mv {} /destination/dir

I learned this line of code from this forum as well and it works fine. However, file movement is kinda slow; about 1-2 files per second. At this rate, it may take days to move the files. I have not much background yet about xargs, so I was wondering if there could be a faster way to accomplish this process.

Here are some more details:
–OS is HP-UX.
-The files in /source/dir are continually being added.
-Size per file is around 300-1000kb.
-Filename pattern includes YYYYMMDD date (might prove useful for batch processing).
-The mv* command is already encountering “arg list too long,” hence the use of find/xargs.
-/source/dir has no sub directories.
-After moving the files, I would later divide/mv then into different dirs corresponding to their YYYYMMDD date.

Hope the above info helps. Any advise would be greatly appreciated as well.

Thank you.

…read more
Source: FULL ARTICLE at The UNIX and Linux Forums

Linux

Date format to be changed from DDMMYYYY to YYYYMMDD

Leave a comment

By djrulz123

My requirement is:- there will be files at a location each day with the date format DDMMYYYY.

Novawise_Activity_Call_Notes_04022013.txt
Novawise_Activity_Inbound_04022013.txt
Novawise_Activity_Inbound_05022013.txt
Novawise_Activity_Call_Notes_05022013.txt
Novawise_Activity_Call_Notes_06022013.txt
Novawise_Activity_Inbound_06022013.txt

Now in another file we store the last run date (LAST_DATE.TXT) in the format 20130405(YYYYMMDD). The current value is 20130405.

I want to pick up only the files for the 6th and not the 4th and the 5th. And then store the date 06022013 (from the file) into the LAST_DATE.TXT file in the format 20130406.
Hence i would process each day only those files which are latest and after the last run date.

Please let me know how this can be done.

…read more
Source: FULL ARTICLE at The UNIX and Linux Forums

Linux

Grep Logs That Are Updating

Leave a comment

By Carl2013

Hello all. I am new to this forum and also very new to using grep – so please excuse me if this question is not in the correct forum and/or is not pertinent to this website.

I use greps like this:

gunzip -c L:System1SailLogger_20071019* L:System2SailLogger_20071019* L:System3SailLogger_20071019* | grep 35=Order

These logs are updating continuosly throughout the day. SailLogger_20071019* is the name of log followed by YYYYMMDD. I use the wildcard to target all the logs (the logs themselves are renamed at the top of each hour – eg 2007101907 (for 7am) and 2007101908 (for 9am).

When I run this grep, it will only extract from the first log in the statement – System1. I then get an error message related to not being able to find the end of the file.

Is there a way to grep all these logs ?

Thanks you for any help and guidance.

Regards.

Source: FULL ARTICLE at The UNIX and Linux Forums