A curiosity with cp and rm - Slackware

This is a discussion on A curiosity with cp and rm - Slackware ; On Feb 7, 7:52 am, Mikhail Zotov wrote: > On Wed, 6 Feb 2008 10:42:30 -0800 (PST) > > B.Yanchit...@gmail.com wrote: > > On Feb 6, 8:32 pm, B.Yanchit...@gmail.com wrote: > > > "rename" is a linux way, true unix ...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 21 to 31 of 31

Thread: A curiosity with cp and rm

  1. Re: A curiosity with cp and rm

    On Feb 7, 7:52 am, Mikhail Zotov wrote:
    > On Wed, 6 Feb 2008 10:42:30 -0800 (PST)
    >
    > B.Yanchit...@gmail.com wrote:
    > > On Feb 6, 8:32 pm, B.Yanchit...@gmail.com wrote:
    > > > "rename" is a linux way, true unix way is
    > > > ls *txt | sed 's/.*/mv & &.old' | sh

    >
    > > Oops, there must be of course (extra "/" for sed)
    > > ls *txt | sed 's/.*/mv & &.old/' | sh

    >
    > An this demonstrates a true way is not necessarily
    > the most effective one :-)
    >
    > --
    > Mikhail


    True way is always the most effective one :-), but there is a price
    for that, instead of single
    command one has to create pipe. What about to move all files with txt
    extension in "A" directory into txt.old in "B" directory? You'll
    quickly find that you have to write script or you might create a pipe,
    something like
    ls A/*txt | sed 's+A/++; s+.*+mv A/& B/&.old+' | sh

    ---
    Bogdan

  2. Re: A curiosity with cp and rm

    Mikhail Zotov wrote:
    > On 07 Feb 2008 07:40:17 GMT
    > Loki Harfagr wrote:
    >
    >> Thu, 07 Feb 2008 08:52:02 +0300, Mikhail Zotov did cat :
    >>
    >>> On Wed, 6 Feb 2008 10:42:30 -0800 (PST) B.Yanchitsky@gmail.com wrote:

    > ....
    >>>> Oops, there must be of course (extra "/" for sed) ls *txt | sed
    >>>> 's/.*/mv & &.old/' | sh
    >>> An this demonstrates a true way is not necessarily the most effective
    >>> one :-)

    > ....
    >> Though I think he meant:
    >> # ls *txt | sed 's/\(^.*\)\(.txt\)/mv \1\2 \1.old/'
    >> mv 1txt.txt 1txt.old
    >> mv 2txt.txt 2txt.old
    >>
    >> which of course gives any player more chances to b0rk the syntax ;D)

    >
    > I'll probably add a bit of spices:
    >
    > alias ls='/bin/ls -1 --color -l'
    >
    > Try the trick then :^)
    >


    Use Steve's method:

    for x in *.txt ; do mv $x ${x%.txt}.old ; done

    or

    for x in *.txt ; do mv $x ${x%.*}.old ; done

    of course, that requires bash, I dunno if it'll work under sh or csh or
    any others. (probably works fine under sh)

    Ray

  3. Re: A curiosity with cp and rm

    On Feb 7, 11:22 am, Mikhail Zotov wrote:
    >
    > > which of course gives any player more chances to b0rk the syntax ;D)

    >
    > I'll probably add a bit of spices:
    >
    > alias ls='/bin/ls -1 --color -l'
    >
    > Try the trick then :^)
    >
    > --
    > Mikhail


    Yeah, alias ls=rm would be quite hot too. I wrote about true way
    because, in my understanding, the unix way is a small collection of
    specific and keen tools, and most of the problems can be solved using
    this small collection. There is another philosophy, a mess of
    instruments with unclear purpose. For example, quite recently, I had
    to remove duplicated lines from an unsorted text file. There is
    "uniq" that removes duplicated lines from sorted(???) file, thus it
    is designed for very very specific and unclear problem. What I was
    looking for is

    cat file| awk '!x[$0]++'.

    Why to remember about "uniq", it would be better to learn awk.

    ---
    Bogdan

  4. Re: A curiosity with cp and rm

    On Thu, 07 Feb 2008 09:14:22 +0000, Tom Newton wrote:

    > On 2008-02-07, Loki Harfagr wrote:
    >> Thu, 07 Feb 2008 08:52:02 +0300, Mikhail Zotov did catÂ*:
    >>
    >>> On Wed, 6 Feb 2008 10:42:30 -0800 (PST) B.Yanchitsky@gmail.com wrote:
    >>>
    >>>> On Feb 6, 8:32 pm, B.Yanchit...@gmail.com wrote:
    >>>> > "rename" is a linux way, true unix way is ls *txt | sed 's/.*/mv &
    >>>> > &.old' | sh
    >>>> >
    >>>> >
    >>>> Oops, there must be of course (extra "/" for sed) ls *txt | sed
    >>>> 's/.*/mv & &.old/' | sh
    >>>
    >>> An this demonstrates a true way is not necessarily the most effective
    >>> one :-)

    >>
    >> And still it was probably not the result he hoped,
    >> let's test on a sample:
    >>
    >> # ls $txt
    >> 1txt.txt 2txt.txt
    >>
    >> Nice try:
    >> # ls *txt | sed 's/.*/mv & &.old/'
    >> mv 1txt.txt 1txt.txt.old
    >> mv 2txt.txt 2txt.txt.old
    >>
    >> Though I think he meant:
    >> # ls *txt | sed 's/\(^.*\)\(.txt\)/mv \1\2 \1.old/' mv 1txt.txt
    >> 1txt.old
    >> mv 2txt.txt 2txt.old
    >>
    >> which of course gives any player more chances to b0rk the syntax ;D)

    >
    > Now that you've fixed it, I must say that that is a clever little
    > script. I always use a for loop if there are a lot of files. Or a file
    > manager or manually if there is only a few. Never tried the rename
    > utility.


    I happen to use the rename sometimes mainly because it wakes me
    up a bit to have to test thrice each time to try and find how
    to use it without bursting out in tears or laughter ;-)

    In my scripts I usually simply use the var subst off the shell
    (e.g.: "${ll//.flac/}.ogg")
    in a loop, note that the 'rename' tool will fail (thru bash limit)
    on heavy dirs stuff like:
    ------------
    $ ls | wc
    50000 50000 2444470

    $ ls |head -n 1
    1.12345678901234567890tttttttt.more.more.more.more .txt
    ------------

    ------------
    $ rename .txt$ .toto *
    -bash: /usr/bin/rename: Argument list too long
    ------------

    while the loop will take its time but work it out fine:
    ------------
    $ time for ll in *.txt; do mv "${ll}" "${ll//.txt/}.toto" ; done

    real 3m9.470s
    user 0m45.618s
    sys 2m21.391s
    ------------

    >
    > I wonder how you'd do that with awk instead of sed.


    I wouldn't, it'd have to use sub or gsub which basically
    are constructs "Ă* la sed" and it'll spoil a call quite the same.
    Use each tool for what it does the best, always mind the proverb
    "messing with files you're bound to get nodes".

    > I hate sed. It gives
    > me these great big HEADACHES. :-\
    >
    > Awk is sort of like bash.


    That's quite a curious way to see them two!
    A bit like saying your BMX is sort of your typewriter :-)

    > I don't know what the hell sed is like.


    sed is like ed+ex, hence it's the vi of the deaf mute and blind keyboard ;-)

  5. Re: A curiosity with cp and rm

    On 2008-02-07, loki harfagr wrote:
    > On Thu, 07 Feb 2008 09:14:22 +0000, Tom Newton wrote:
    >
    >> On 2008-02-07, Loki Harfagr wrote:
    >>> Thu, 07 Feb 2008 08:52:02 +0300, Mikhail Zotov did cat*:
    >>>
    >>>> On Wed, 6 Feb 2008 10:42:30 -0800 (PST) B.Yanchitsky@gmail.com wrote:
    >>>>
    >>>>> On Feb 6, 8:32 pm, B.Yanchit...@gmail.com wrote:
    >>>>> > "rename" is a linux way, true unix way is ls *txt | sed 's/.*/mv &
    >>>>> > &.old' | sh
    >>>>> >
    >>>>> >
    >>>>> Oops, there must be of course (extra "/" for sed) ls *txt | sed
    >>>>> 's/.*/mv & &.old/' | sh
    >>>>
    >>>> An this demonstrates a true way is not necessarily the most effective
    >>>> one :-)
    >>>
    >>> And still it was probably not the result he hoped,
    >>> let's test on a sample:
    >>>
    >>> # ls $txt
    >>> 1txt.txt 2txt.txt
    >>>
    >>> Nice try:
    >>> # ls *txt | sed 's/.*/mv & &.old/'
    >>> mv 1txt.txt 1txt.txt.old
    >>> mv 2txt.txt 2txt.txt.old
    >>>
    >>> Though I think he meant:
    >>> # ls *txt | sed 's/\(^.*\)\(.txt\)/mv \1\2 \1.old/' mv 1txt.txt
    >>> 1txt.old
    >>> mv 2txt.txt 2txt.old
    >>>
    >>> which of course gives any player more chances to b0rk the syntax ;D)

    >>
    >> Now that you've fixed it, I must say that that is a clever little
    >> script. I always use a for loop if there are a lot of files. Or a file
    >> manager or manually if there is only a few. Never tried the rename
    >> utility.

    >
    > I happen to use the rename sometimes mainly because it wakes me
    > up a bit to have to test thrice each time to try and find how
    > to use it without bursting out in tears or laughter ;-)


    :-) A fact of life!

    >
    > In my scripts I usually simply use the var subst off the shell
    > (e.g.: "${ll//.flac/}.ogg")
    > in a loop, note that the 'rename' tool will fail (thru bash limit)
    > on heavy dirs stuff like:
    > ------------
    > $ ls | wc
    > 50000 50000 2444470
    >
    > $ ls |head -n 1
    > 1.12345678901234567890tttttttt.more.more.more.more .txt
    > ------------
    >
    > ------------
    > $ rename .txt$ .toto *
    > -bash: /usr/bin/rename: Argument list too long
    > ------------
    >
    > while the loop will take its time but work it out fine:
    > ------------
    > $ time for ll in *.txt; do mv "${ll}" "${ll//.txt/}.toto" ; done
    >
    > real 3m9.470s
    > user 0m45.618s
    > sys 2m21.391s
    > ------------


    Sure. That makes sense. I use basename/dirname.

    >
    >>
    >> I wonder how you'd do that with awk instead of sed.

    >
    > I wouldn't, it'd have to use sub or gsub which basically
    > are constructs "à la sed" and it'll spoil a call quite the same.
    > Use each tool for what it does the best, always mind the proverb
    > "messing with files you're bound to get nodes".
    >
    >> I hate sed. It gives
    >> me these great big HEADACHES. :-\
    >>
    >> Awk is sort of like bash.

    >
    > That's quite a curious way to see them two!
    > A bit like saying your BMX is sort of your typewriter :-)
    >
    >> I don't know what the hell sed is like.

    >
    > sed is like ed+ex, hence it's the vi of the deaf mute and blind keyboard ;-)


    Yeh. And I have no problem with the sed scripts that are like
    commands you do at the vi prompt. It's when you get into the H
    and N (etc.) commands with sed that it gets weird.

    I suppose I should just do like you suggest, and use sed for the simple
    stuff and awk for the more complex operations.

    Tom

    --
    calhobbit (at)
    gmail [DOT] com

  6. Re: A curiosity with cp and rm

    Tom Newton wrote:
    >
    > Yeh. And I have no problem with the sed scripts that are like
    > commands you do at the vi prompt. It's when you get into the H
    > and N (etc.) commands with sed that it gets weird.
    >
    > I suppose I should just do like you suggest, and use sed for the simple
    > stuff and awk for the more complex operations.
    >
    > Tom
    >


    and this is the beauty and nightmare of Linux. Choice. Even though the
    unix way was one tool for one job, it seems we somehow ended up with
    many tools for many jobs.

    to rename files you can use:
    mv
    rename
    sed
    awk
    perl
    bash
    I'm sure you can do it in python.
    Someone probably has a makefile for it...
    I bet you can even do it in KDE if you have lots of horsepower.

    I'm not saying it's bad. Choice is good. Although, sometimes even I
    wish it was just ONE way. But... it's not like when *nix started there
    was 10 ways of doing it, it's evolved that way.



  7. Re: A curiosity with cp and rm

    On 07 Feb 2008 18:41:44 GMT
    loki harfagr wrote:

    ....
    > ------------
    > $ ls | wc
    > 50000 50000 2444470
    >
    > $ ls |head -n 1
    > 1.12345678901234567890tttttttt.more.more.more.more .txt
    > ------------
    >
    > ------------
    > $ rename .txt$ .toto *
    > -bash: /usr/bin/rename: Argument list too long
    > ------------


    Not actually being an advocate of `rename', I wonder
    what happens if you run

    $ rename .txt$ .toto *.txt

    --
    Mikhail

  8. Re: A curiosity with cp and rm

    On Fri, 08 Feb 2008 10:29:33 +0300, Mikhail Zotov wrote:

    > On 07 Feb 2008 18:41:44 GMT
    > loki harfagr wrote:
    >
    > ...
    >> ------------
    >> $ ls | wc
    >> 50000 50000 2444470
    >>
    >> $ ls |head -n 1
    >> 1.12345678901234567890tttttttt.more.more.more.more .txt ------------
    >>
    >> ------------
    >> $ rename .txt$ .toto *
    >> -bash: /usr/bin/rename: Argument list too long ------------

    >
    > Not actually being an advocate of `rename', I wonder what happens if you
    > run
    >
    > $ rename .txt$ .toto *.txt


    Well, quite the same as I would've say given the test set I used
    but at least it's easy to test:
    --------
    $ mkdir /dev/shm/DUMP && cd /dev/shm/DUMP && for i in $(seq 50000); do touch ${i}"12345678901234567890tttttttt.more.more.more.more.t xt"; done
    $ rename .txt$ .toto *
    -bash: /usr/bin/rename: Argument list too long
    $ rename .txt$ .toto *.txt
    -bash: /usr/bin/rename: Argument list too long
    --------

    I guess you wanted to know about the case there were too
    many files overall in the dir but the subset of *.txt files
    was much smaller, it goes fine:
    --------
    $ rename .txt .toto 3*.txt
    ....
    $ ls *.toto | tail -n 1
    3999912345678901234567890tttttttt.more.more.more.m ore.toto
    --------

  9. Re: A curiosity with cp and rm

    On 08 Feb 2008 09:41:29 GMT
    loki harfagr wrote:

    > On Fri, 08 Feb 2008 10:29:33 +0300, Mikhail Zotov wrote:
    >

    ....
    > > Not actually being an advocate of `rename', I wonder what happens if you
    > > run
    > >
    > > $ rename .txt$ .toto *.txt

    >
    > Well, quite the same as I would've say given the test set I used
    > but at least it's easy to test:
    > --------
    > $ mkdir /dev/shm/DUMP && cd /dev/shm/DUMP && for i in $(seq 50000); do touch ${i}"12345678901234567890tttttttt.more.more.more.more.t xt"; done
    > $ rename .txt$ .toto *
    > -bash: /usr/bin/rename: Argument list too long
    > $ rename .txt$ .toto *.txt
    > -bash: /usr/bin/rename: Argument list too long
    > --------
    >
    > I guess you wanted to know about the case there were too
    > many files overall in the dir but the subset of *.txt files
    > was much smaller, it goes fine:
    > --------
    > $ rename .txt .toto 3*.txt
    > ...
    > $ ls *.toto | tail -n 1
    > 3999912345678901234567890tttttttt.more.more.more.m ore.toto
    > --------


    Thank you, I see. It seems we have arrived at the limitation
    of the kernel pointed by Thomas Overgaard.

    Cheers,
    Mikhail

  10. Re: A curiosity with cp and rm

    Pipe vs. Loop

    I've made some simple test, what is more efficient pipe or loop?
    Firstly we create a directory containing 100000 files.

    mkdir test_p && cd test_p
    for (( i = 0 ; i < 100000 ; i++ ))
    do
    touch file$i.txt
    done
    exit

    Next we create pipe. I created some quite nice pipe, that renames only
    files and does not touch directories. The pipe replaces "txt" by "old"
    at the tail of file name. ls *txt does not work, argument list too
    long.
    find . -not -type d -name '*txt' -maxdepth 1 | sed 's/txt$//; s/.*/mv
    &txt &old/'|sh

    Next is bash loop
    for i in *txt
    do
    mv $i ${i%txt}old
    done

    I decided not to fight in bash script for renaming only files and
    omitting directories.

    Time for pipe
    19.137u 45.638s 1:08.16 95.0% 0+0k 0+0io 0pf+0w
    Time for loop
    11.416u 98.774s 2:21.63 77.7% 0+0k 0+0io 0pf+0w

    i.e. time for pipe is 1 min. 08 sec., time for loop 2 min. 21 sec.

    Unfortunately it's difficult to obtain memory usage, probably "time"
    does not understand format string.
    What I've seen using ps xau during run, for pipe: find ~ 6Mb, sed ~
    2Mb.
    For bash loop, bash explodes ~ 15 Mb, probably it has to store *txt
    in the RAM.

    Thus, pipe is significantly faster and consumes less memory, but more
    CPU usage. Probably because pipe is completely unrolled loop, that is
    executed line by line, and there is no need to store all input data.
    But to create pipes one has to understand quite well what the regular
    expressions are. Flexible and keen tool, but you might easily cut your
    fingers off.
    Bash, quite clear syntax, and may be more safe than pipe.

    ---
    Bogdan

  11. Re: A curiosity with cp and rm

    On Feb 8, 3:20 am, Tom Newton wrote:
    > Yeh. And I have no problem with the sed scripts that are like
    > commands you do at the vi prompt. It's when you get into the H
    > and N (etc.) commands with sed that it gets weird.
    >
    > I suppose I should just do like you suggest, and use sed for the simple
    > stuff and awk for the more complex operations.
    >
    > Tom


    Take attention to what Mikhail Zotov wrote, about aliased ls.
    Quite conservative Slackware provides you its "slackware" version of
    ls.
    Idea, of course, bells and whistles
    $alias
    ls /bin/ls $LS_OPTIONS

    but ls is very basic command, and some scripts, pipes might be broken.

    Thus, to be safe, it would be better at first to unalias everything in
    profile.

    ---
    Bogdan


+ Reply to Thread
Page 2 of 2 FirstFirst 1 2