A curiosity with cp and rm - Slackware

This is a discussion on A curiosity with cp and rm - Slackware ; Last night I had to copy over 7,000 files from one directory to another. I cd to the source directory and cp * /to-the-destination-directory. I've done this thousand time with other file groups. Interestingly, I got an error message.. 'argument ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 31

Thread: A curiosity with cp and rm

  1. A curiosity with cp and rm

    Last night I had to copy over 7,000 files from one directory to another. I
    cd to the source directory and cp * /to-the-destination-directory. I've
    done this thousand time with other file groups.

    Interestingly, I got an error message.. 'argument list too long'... all
    the files have a ztxt extension. After some time, I tried cp *.ztxt
    /to-the-destination-directory, that worked. The same with rm when I had to
    delete the source files.

    Just idle curiosity, any idea why.

    It's still the same argument list, it's still 7,000 plus files.

    ken

  2. Re: A curiosity with cp and rm

    No_One wrote:
    > Last night I had to copy over 7,000 files from one directory to another. I
    > cd to the source directory and cp * /to-the-destination-directory. I've
    > done this thousand time with other file groups.
    >
    > Interestingly, I got an error message.. 'argument list too long'... all
    > the files have a ztxt extension. After some time, I tried cp *.ztxt
    > /to-the-destination-directory, that worked. The same with rm when I had to
    > delete the source files.
    >
    > Just idle curiosity, any idea why.
    >
    > It's still the same argument list, it's still 7,000 plus files.


    My guess is that there is something you are overlooking. the argument
    list is develop by the shell. Is there not some backup files (ending
    with ~)that are added when you do cp * ? try echo *.ztxt > ~/ztxt-files.
    then echo * > ~/all-files and compare the files (If you do this the echo
    would be that of the shell so that you won't get the argument list too
    long error messages).

    Olive

  3. Re: A curiosity with cp and rm

    Olive wrote:
    > No_One wrote:
    >> Last night I had to copy over 7,000 files from one directory to
    >> another. I
    >> cd to the source directory and cp * /to-the-destination-directory. I've
    >> done this thousand time with other file groups.
    >>
    >> Interestingly, I got an error message.. 'argument list too long'... all
    >> the files have a ztxt extension. After some time, I tried cp *.ztxt
    >> /to-the-destination-directory, that worked. The same with rm when I
    >> had to
    >> delete the source files.
    >>
    >> Just idle curiosity, any idea why.
    >>
    >> It's still the same argument list, it's still 7,000 plus files.

    >
    > My guess is that there is something you are overlooking. the argument
    > list is develop by the shell. Is there not some backup files (ending
    > with ~)that are added when you do cp * ? try echo *.ztxt > ~/ztxt-files.
    > then echo * > ~/all-files and compare the files (If you do this the echo
    > would be that of the shell so that you won't get the argument list too
    > long error messages).


    Try also (in bash)

    shopt -s extglob
    echo !(*.ztxt)

    to find all file that are not ending with ztxt. My guess is that you
    will find some.

    Olive

  4. Re: A curiosity with cp and rm

    On 2008-02-05, Olive wrote:
    > No_One wrote:
    >
    > My guess is that there is something you are overlooking. the argument
    > list is develop by the shell. Is there not some backup files (ending
    > with ~)that are added when you do cp * ? try echo *.ztxt > ~/ztxt-files.
    > then echo * > ~/all-files and compare the files (If you do this the echo
    > would be that of the shell so that you won't get the argument list too
    > long error messages).
    >
    > Olive


    The .ztxt file are generated from the contents of a mysql table. The
    directory has to be empty before it will run to create the contents, if
    files exist the program, actually php script, will terminate before
    generating any files.

    That said, I did find *|wc -l, then compared to the results of
    find *.ztxt|wc -l...the same number of files with each execution.

    ken

  5. Re: A curiosity with cp and rm

    No_One wrote:
    > Last night I had to copy over 7,000 files from one directory to another. I
    > cd to the source directory and cp * /to-the-destination-directory. I've
    > done this thousand time with other file groups.
    >
    > Interestingly, I got an error message.. 'argument list too long'... all
    > the files have a ztxt extension. After some time, I tried cp *.ztxt
    > /to-the-destination-directory, that worked. The same with rm when I had to
    > delete the source files.
    >
    > Just idle curiosity, any idea why.
    >
    > It's still the same argument list, it's still 7,000 plus files.
    >
    > ken


    does the shell parse
    cp *
    differently than
    cp *.ztxt
    ?

    For example, in DOS, I got used to typing
    ren *.txt *.old
    and if you try
    mv *.txt *.old
    with bash, you'll not get the result you seek.

    I know that xargs exists because of this - cp * ends up being cp
    file1.txt file2.txt etc...

    I guess what I'm saying is that your command line is too long one way
    and not the other, but I don't even know how to test that. Maybe a bash
    guru can step in here and explain it better.

    Ray

  6. Re: A curiosity with cp and rm


    No One wrote :

    > Just idle curiosity, any idea why.


    Its a limitation in the kernel and the error message comes from bash:


    Dunno why cp *.ztxt doesn't fail, maybe its the new toolchain.
    --
    Thomas O.

    This area is designed to become quite warm during normal operation.

  7. Re: A curiosity with cp and rm

    On Tue, 5 Feb 2008, No_One wrote:

    >
    > Last night I had to copy over 7,000 files from one directory to another. I
    > cd to the source directory and cp * /to-the-destination-directory. I've
    > done this thousand time with other file groups.
    >
    > Interestingly, I got an error message.. 'argument list too long'... all
    > the files have a ztxt extension. After some time, I tried cp *.ztxt
    > /to-the-destination-directory, that worked. The same with rm when I had to
    > delete the source files.
    >
    > Just idle curiosity, any idea why.


    Yes, its the limitation on the shell you are using, it might be 7K files
    but the 'list' may exceed around 32K in size, use xargs to get
    around this.


    --
    Cheers
    Res

    mysql> update auth set Framed-IP-Address='127.0.0.127' where user= 'troll';

  8. Re: A curiosity with cp and rm

    On Tue, 05 Feb 2008 21:09:24 GMT
    Ray wrote:

    > For example, in DOS, I got used to typing
    > ren *.txt *.old
    > and if you try
    > mv *.txt *.old
    > with bash, you'll not get the result you seek.


    rename .txt .old *.txt

    --
    Mikahil

  9. Re: A curiosity with cp and rm

    On 2008-02-05, Thomas Overgaard wrote:
    >
    > No One wrote :
    >
    >> Just idle curiosity, any idea why.

    >
    > Its a limitation in the kernel and the error message comes from bash:
    >
    >
    > Dunno why cp *.ztxt doesn't fail, maybe its the new toolchain.


    Thanks for the link, that answers at least part of the question. I suspect,
    and this is based on my limited C experience - I stress the term 'limited C
    experience', maybe with cp * cp has to create the file list with extensions
    included and with cp *.ztxt, it only has to record the extension once????

    Maybe, maybe not...

    Regardless, thanks.

    ken

  10. Re: A curiosity with cp and rm

    On 2008-02-05, Res wrote:
    > On Tue, 5 Feb 2008, No_One wrote:
    >
    >>
    >> Last night I had to copy over 7,000 files from one directory to another. I
    >> cd to the source directory and cp * /to-the-destination-directory. I've
    >> done this thousand time with other file groups.
    >>
    >> Interestingly, I got an error message.. 'argument list too long'... all
    >> the files have a ztxt extension. After some time, I tried cp *.ztxt
    >> /to-the-destination-directory, that worked. The same with rm when I had to
    >> delete the source files.
    >>
    >> Just idle curiosity, any idea why.

    >
    > Yes, its the limitation on the shell you are using, it might be 7K files
    > but the 'list' may exceed around 32K in size, use xargs to get
    > around this.
    >
    >


    Yeah, xarg would be a solution, one that is in fact recommended.

    Thanks for the info.

    ken

  11. Re: A curiosity with cp and rm

    On 2008-02-05, Ray wrote:
    >
    > does the shell parse
    > cp *
    > differently than
    > cp *.ztxt
    > ?
    >
    > For example, in DOS, I got used to typing
    > ren *.txt *.old
    > and if you try
    > mv *.txt *.old
    > with bash, you'll not get the result you seek.


    It may parse thing differently, not sure. However, as someone poited ot
    xarg should solve the problem

    thanks

    ken

  12. Re: A curiosity with cp and rm

    Mikhail Zotov wrote:
    > On Tue, 05 Feb 2008 21:09:24 GMT
    > Ray wrote:
    >
    >> For example, in DOS, I got used to typing
    >> ren *.txt *.old
    >> and if you try
    >> mv *.txt *.old
    >> with bash, you'll not get the result you seek.

    >
    > rename .txt .old *.txt
    >
    > --
    > Mikahil

    learn something new every day...

    I was just usually doing something like

    for x in *.txt ; do mv $x $x.old ; done

    (but then you end up with filename.txt becoming filename.txt.old)

    Ray

  13. Re: A curiosity with cp and rm

    On Feb 6, 5:31 pm, ray wrote:
    >
    > learn something new every day...
    >
    > I was just usually doing something like
    >
    > for x in *.txt ; do mv $x $x.old ; done
    >
    > (but then you end up with filename.txt becoming filename.txt.old)
    >
    > Ray


    "rename" is a linux way, true unix way is
    ls *txt | sed 's/.*/mv & &.old' | sh

    This construction is very flexible and may be useful for other
    purposes too.

    ---
    Bogdan



  14. Re: A curiosity with cp and rm

    On Feb 6, 8:32 pm, B.Yanchit...@gmail.com wrote:
    > "rename" is a linux way, true unix way is
    > ls *txt | sed 's/.*/mv & &.old' | sh
    >


    Oops, there must be of course (extra "/" for sed)
    ls *txt | sed 's/.*/mv & &.old/' | sh

    ---
    Bogdan


  15. Re: A curiosity with cp and rm

    ray wrote:
    > Mikhail Zotov wrote:
    >> On Tue, 05 Feb 2008 21:09:24 GMT
    >> Ray wrote:
    >>
    >>> For example, in DOS, I got used to typing
    >>> ren *.txt *.old
    >>> and if you try
    >>> mv *.txt *.old
    >>> with bash, you'll not get the result you seek.

    >>
    >> rename .txt .old *.txt
    >>
    >> --
    >> Mikahil

    > learn something new every day...
    >
    > I was just usually doing something like
    >
    > for x in *.txt ; do mv $x $x.old ; done
    >
    > (but then you end up with filename.txt becoming filename.txt.old)


    for x in *.txt ; do mv $x ${x%.txt}.old ; done

    or

    for x in *.txt ; do mv $x ${x%.*}.old ; done

    Steve

  16. Re: A curiosity with cp and rm

    On Wed, 6 Feb 2008 10:42:30 -0800 (PST)
    B.Yanchitsky@gmail.com wrote:

    > On Feb 6, 8:32 pm, B.Yanchit...@gmail.com wrote:
    > > "rename" is a linux way, true unix way is
    > > ls *txt | sed 's/.*/mv & &.old' | sh
    > >

    >
    > Oops, there must be of course (extra "/" for sed)
    > ls *txt | sed 's/.*/mv & &.old/' | sh


    An this demonstrates a true way is not necessarily
    the most effective one :-)

    --
    Mikhail

  17. Re: A curiosity with cp and rm

    Thu, 07 Feb 2008 08:52:02 +0300, Mikhail Zotov did catĀ*:

    > On Wed, 6 Feb 2008 10:42:30 -0800 (PST) B.Yanchitsky@gmail.com wrote:
    >
    >> On Feb 6, 8:32 pm, B.Yanchit...@gmail.com wrote:
    >> > "rename" is a linux way, true unix way is ls *txt | sed 's/.*/mv &
    >> > &.old' | sh
    >> >
    >> >

    >> Oops, there must be of course (extra "/" for sed) ls *txt | sed
    >> 's/.*/mv & &.old/' | sh

    >
    > An this demonstrates a true way is not necessarily the most effective
    > one :-)


    And still it was probably not the result he hoped,
    let's test on a sample:

    # ls $txt
    1txt.txt 2txt.txt

    Nice try:
    # ls *txt | sed 's/.*/mv & &.old/'
    mv 1txt.txt 1txt.txt.old
    mv 2txt.txt 2txt.txt.old

    Though I think he meant:
    # ls *txt | sed 's/\(^.*\)\(.txt\)/mv \1\2 \1.old/'
    mv 1txt.txt 1txt.old
    mv 2txt.txt 2txt.old

    which of course gives any player more chances to b0rk the syntax ;D)

  18. Re: A curiosity with cp and rm

    On 2008-02-07, Loki Harfagr wrote:
    > Thu, 07 Feb 2008 08:52:02 +0300, Mikhail Zotov did cat*:
    >
    >> On Wed, 6 Feb 2008 10:42:30 -0800 (PST) B.Yanchitsky@gmail.com wrote:
    >>
    >>> On Feb 6, 8:32 pm, B.Yanchit...@gmail.com wrote:
    >>> > "rename" is a linux way, true unix way is ls *txt | sed 's/.*/mv &
    >>> > &.old' | sh
    >>> >
    >>> >
    >>> Oops, there must be of course (extra "/" for sed) ls *txt | sed
    >>> 's/.*/mv & &.old/' | sh

    >>
    >> An this demonstrates a true way is not necessarily the most effective
    >> one :-)

    >
    > And still it was probably not the result he hoped,
    > let's test on a sample:
    >
    > # ls $txt
    > 1txt.txt 2txt.txt
    >
    > Nice try:
    > # ls *txt | sed 's/.*/mv & &.old/'
    > mv 1txt.txt 1txt.txt.old
    > mv 2txt.txt 2txt.txt.old
    >
    > Though I think he meant:
    > # ls *txt | sed 's/\(^.*\)\(.txt\)/mv \1\2 \1.old/'
    > mv 1txt.txt 1txt.old
    > mv 2txt.txt 2txt.old
    >
    > which of course gives any player more chances to b0rk the syntax ;D)


    Now that you've fixed it, I must say that that is a clever little script. I always use
    a for loop if there are a lot of files. Or a file manager or manually if there is only
    a few. Never tried the rename utility.

    I wonder how you'd do that with awk instead of sed. I hate sed. It gives me these
    great big HEADACHES. :-\

    Awk is sort of like bash. I don't know what the hell sed is like.

    Tom


    --
    calhobbit (at)
    gmail [DOT] com

  19. Re: A curiosity with cp and rm

    On 07 Feb 2008 07:40:17 GMT
    Loki Harfagr wrote:

    > Thu, 07 Feb 2008 08:52:02 +0300, Mikhail Zotov did catš:
    >
    > > On Wed, 6 Feb 2008 10:42:30 -0800 (PST) B.Yanchitsky@gmail.com wrote:

    ....
    > >> Oops, there must be of course (extra "/" for sed) ls *txt | sed
    > >> 's/.*/mv & &.old/' | sh

    > >
    > > An this demonstrates a true way is not necessarily the most effective
    > > one :-)

    ....
    > Though I think he meant:
    > # ls *txt | sed 's/\(^.*\)\(.txt\)/mv \1\2 \1.old/'
    > mv 1txt.txt 1txt.old
    > mv 2txt.txt 2txt.old
    >
    > which of course gives any player more chances to b0rk the syntax ;D)


    I'll probably add a bit of spices:

    alias ls='/bin/ls -1 --color -l'

    Try the trick then :^)

    --
    Mikhail

  20. Re: A curiosity with cp and rm

    On Feb 7, 9:40 am, Loki Harfagr wrote:
    > And still it was probably not the result he hoped,
    > Though I think he meant:
    > # ls *txt | sed 's/\(^.*\)\(.txt\)/mv \1\2 \1.old/'
    > mv 1txt.txt 1txt.old
    > mv 2txt.txt 2txt.old
    >
    > which of course gives any player more chances to b0rk the syntax ;D)


    Another and simpler version to replace "txt" with "old"

    ls *txt | sed 's/txt$//; s/.*/mv &txt &old/'|sh

    You might replace ls with find, sed with awk or pearl, but in any case
    one has to know either sed or perl or awk.
    For example to perform "make clean" in all subdirectories with
    Makefile
    find . -name 'Makefile' -print | sed 's/Makefile$//; s/.*/(cd &; make
    clean)/'|sh

    When creating such constructions I typically replace, for example,
    "make clean" with pwd, or omit |sh on the tail, to see what will be
    executed.

    ---
    Bogdan

+ Reply to Thread
Page 1 of 2 1 2 LastLast