virtual address space size of a proces - HP UX

This is a discussion on virtual address space size of a proces - HP UX ; Hi, I am looking to work on unix systems. I want to get the total virtual address space of a process, the used virtual memory of a process i am able to get without any problem. I have tried using ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 23

Thread: virtual address space size of a proces

  1. virtual address space size of a proces

    Hi,

    I am looking to work on unix systems. I want to get the total virtual
    address space of a process, the used virtual memory of a process i am
    able to get without any problem. I have tried using getrlimit but that
    gives only RLIMIT_INIFINITY or RLIMIT64_INFINITY if i use the
    getrlimit64.

    Can you give me an idea on how much virtual addressable space is there
    in each respective unix system for a user address space as i need to
    do my memory allocations based on that and show a warning in an
    application when the memory is about to get depleted?


    Best regards,
    RUI

  2. Re: virtual address space size of a proces

    On Nov 4, 9:25*am, guidevelo...@gmail.com wrote:
    > Hi,
    >
    > I am looking to work on unix systems. I want to get the total virtual
    > address space of a process, the used virtual memory of a process i am
    > able to get without any problem. I have tried using getrlimit but that
    > gives only *RLIMIT_INIFINITY or RLIMIT64_INFINITY if i use the
    > getrlimit64.
    >
    > Can you give me an idea on how much virtual addressable space is there
    > in each respective unix system for a user address space as i need to
    > do my memory allocations based on that and show a warning in an
    > application when the memory is about to get depleted?
    >
    > Best regards,
    > RUI


    I'm not sure you can do much better than 2^sizeof(void *).

    DS

  3. Re: virtual address space size of a proces

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    David Schwartz wrote:
    > I'm not sure you can do much better than 2^sizeof(void *).


    Even that is wrong:
    1. sizeof returns size in bytes. To get amount of memory that *CPU* can
    access you have to get pointer's size in bits,
    2. memory accessible for *CPU* (and kernel in that case) != memory
    accessible for user space program.

    Pawel Dziepak
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)
    Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

    iEYEARECAAYFAkkQvWMACgkQPFW+cUiIHNo6ZwCghDNSnUXy83 feFL8jsjyALuBm
    bKEAn0wDCnGPaKlY6c3aELk6kZV4LCAF
    =Qkry
    -----END PGP SIGNATURE-----

  4. Re: virtual address space size of a proces

    On Nov 4, 9:47*pm, and...@cucumber.demon.co.uk (Andrew Gabriel) wrote:
    > In article <53671ae0-eeb2-4d28-83f6-4073a5742...@w1g2000prk.googlegroups.com>,
    > * * * * guidevelo...@gmail.com writes:
    >
    > > Hi,

    >
    > > I am looking to work on unix systems. I want to get the total virtual
    > > address space of a process, the used virtual memory of a process i am
    > > able to get without any problem. I have tried using getrlimit but that
    > > gives only *RLIMIT_INIFINITY or RLIMIT64_INFINITY if i use the
    > > getrlimit64.

    >
    > > Can you give me an idea on how much virtual addressable space is there
    > > in each respective unix system for a user address space as i need to
    > > do my memory allocations based on that and show a warning in an
    > > application when the memory is about to get depleted?

    >
    > For a 32 bit process, it's going to be 4GB minus an ammount
    > which depends on the OS and CPU. Some CPU's run the kernel
    > in the same address space although hidden from the user-land
    > process (e.g. sun4c, sun4m, sun4d, x86) and have to allow
    > space for the kernel (256MB or 512MB, or 1GB in the case of
    > Windows). Others have a completely separate kernel address
    > space (sun4u, sun4v) so you've got the full 4GB available.
    > There may still be some OS/CPU combinations which impose
    > smaller address spaces due to limitations in MMU's page
    > tables, etc, but with most OS/CPU's being 64 bit nowadays,
    > they don't normally have any problem providing the full
    > address space of a 32 bit application, as what's small
    > compared with what they're expected to deal with.
    >
    > In principle, the same applies for 64 bit processes (but with
    > 1.8E9 GB address space) but in practice I doubt any OS/CPU can
    > setup tables to address that, so the limit is going to be down
    > to the particular OS/CPU combination.
    >
    > --
    > Andrew Gabriel
    > [email address is not usable -- followup in the newsgroup]


    Hi,

    I understand that the address space is 4 GB which could be used by
    kernel space or maybe completely used by user space, however i would
    like to know for solaris 2.4 ownwards, how can i programatically get
    the virtual memory available to the process so that i can show a
    warning to the user when further allocation is not possible.

    Same goes for hp ux 10 if somebody can help?
    And also for suse linux 9 and redhat 9 ownwards?

    Isn't there a common api or a configuration file where i can retrieve
    this information from, for most of the unixes?

    Best regards,
    RUI

  5. Re: virtual address space size of a proces

    guideveloper@gmail.com wrote:
    > the used virtual memory of a process i am
    > able to get without any problem.


    How can you get this for HP-UX?

    > I have tried using getrlimit but that gives only RLIMIT_INIFINITY


    getrlimit(2) works for HP-UX. It returns either ulimit or maxdsiz(5).

    > Same goes for hp-ux 10 if somebody can help?


    This is no longer supported. Only 11.11 and up.

  6. Re: virtual address space size of a proces

    guideveloper@gmail.com writes:

    > Hi,
    >
    > I understand that the address space is 4 GB which could be used by
    > kernel space or maybe completely used by user space, however i would
    > like to know for solaris 2.4 ownwards, how can i programatically get
    > the virtual memory available to the process so that i can show a
    > warning to the user when further allocation is not possible.
    >
    > Same goes for hp ux 10 if somebody can help?
    > And also for suse linux 9 and redhat 9 ownwards?
    >
    > Isn't there a common api or a configuration file where i can retrieve
    > this information from, for most of the unixes?


    On a modern operating system, there really isn't a good way to find out
    "how much more memory can I allocate before I run out". The amount of
    memory available to your process depends on many factors, and is
    constantly changing based on what else is happening in the system.

    Really, about the only thing you can reliably do is to try the
    allocation and see if it succeeds. You can try to write the program in
    such a way that it will behave gracefully when this occurs, perhaps
    ask the user to free up some memory and then try to continue.

    Perhaps you are trying to tune the amount of memory used by your program
    based on how much is available. A better way is to ask the user to
    specify how much to use, since the user might have a better idea of how
    much memory is present, and how much is needed for other tasks, present
    and future. In my experience, programs that try to auto-detect the
    right thing to do usually get it wrong, are hard to understand because
    their behavior keeps changing, and are generally annoying to use.


  7. Re: virtual address space size of a proces

    On 2008-11-05 00:58:48 +0000, Nate Eldredge said:

    > guideveloper@gmail.com writes:
    >
    >> Hi,
    >>
    >> I understand that the address space is 4 GB which could be used by
    >> kernel space or maybe completely used by user space, however i would
    >> like to know for solaris 2.4 ownwards, how can i programatically get
    >> the virtual memory available to the process so that i can show a
    >> warning to the user when further allocation is not possible.
    >>
    >> Same goes for hp ux 10 if somebody can help?
    >> And also for suse linux 9 and redhat 9 ownwards?
    >>
    >> Isn't there a common api or a configuration file where i can retrieve
    >> this information from, for most of the unixes?

    >
    > On a modern operating system, there really isn't a good way to find out
    > "how much more memory can I allocate before I run out". The amount of
    > memory available to your process depends on many factors, and is
    > constantly changing based on what else is happening in the system.
    >
    > Really, about the only thing you can reliably do is to try the
    > allocation and see if it succeeds. You can try to write the program in
    > such a way that it will behave gracefully when this occurs, perhaps
    > ask the user to free up some memory and then try to continue.


    That is usually the right approach, but AIUI it falls apart slightly on
    systems which overcommit VM, like Linux. That is, they return memory to
    the process even if there isn't enough VM. When you try to use the
    returned memory... either you get lucky and the system's got enough VM
    again or the OS has to kill some other poor process to make some VM for
    you.

    I've a feeling HP-UX overcommits VM too.
    --
    Chris


  8. Re: virtual address space size of a proces

    Chris Ridd wrote:
    > I've a feeling HP-UX overcommits VM too.


    No, not by default.

  9. Re: virtual address space size of a proces

    Chris Ridd writes:
    > On 2008-11-05 00:58:48 +0000, Nate Eldredge said:


    [...]

    >> Really, about the only thing you can reliably do is to try the
    >> allocation and see if it succeeds.


    [...]

    > That is usually the right approach, but AIUI it falls apart slightly
    > on systems which overcommit VM, like Linux. That is, they return
    > memory to the process even if there isn't enough VM. When you try to
    > use the returned memory... either you get lucky and the system's got
    > enough VM again or the OS has to kill some other poor process to make
    > some VM for you.


    The default is to kill the process which tried the memory access that
    could not succeed because there wasn't enough virtual memory
    available. But any other behaviour would not 'magically' cause this
    set of processes whose attempts to use the ressources of the system in
    particular way at that time to be more successful.


  10. Re: virtual address space size of a proces

    guideveloper@gmail.com writes:

    [...]

    > I understand that the address space is 4 GB which could be used by
    > kernel space or maybe completely used by user space, however i would
    > like to know for solaris 2.4 ownwards, how can i programatically get
    > the virtual memory available to the process so that i can show a
    > warning to the user when further allocation is not possible.


    The answer is 'you cannot'.

  11. Re: virtual address space size of a proces

    On 2008-11-05 10:39:03 +0000, Rainer Weikusat said:

    > Chris Ridd writes:
    >> On 2008-11-05 00:58:48 +0000, Nate Eldredge said:

    >
    > [...]
    >
    >>> Really, about the only thing you can reliably do is to try the
    >>> allocation and see if it succeeds.

    >
    > [...]
    >
    >> That is usually the right approach, but AIUI it falls apart slightly
    >> on systems which overcommit VM, like Linux. That is, they return
    >> memory to the process even if there isn't enough VM. When you try to
    >> use the returned memory... either you get lucky and the system's got
    >> enough VM again or the OS has to kill some other poor process to make
    >> some VM for you.

    >
    > The default is to kill the process which tried the memory access that
    > could not succeed because there wasn't enough virtual memory
    > available. But any other behaviour would not 'magically' cause this
    > set of processes whose attempts to use the ressources of the system in
    > particular way at that time to be more successful.


    I had in mind the infamous Linux "OOM killer" algorithm, which *will*
    AIUI kill other things.
    --
    Chris


  12. Re: virtual address space size of a proces

    On 5 Nov, 06:11, Chris Ridd wrote:
    > On 2008-11-05 00:58:48 +0000, Nate Eldredge said:
    >
    >
    >
    >
    >
    > > guidevelo...@gmail.com writes:

    >
    > >> Hi,

    >
    > >> I understand that the address space is 4 GB which could be used by
    > >> kernel space or maybe completely used by user space, however i would
    > >> like to know for solaris 2.4 ownwards, how can i programatically get
    > >> the virtual memory available to the process so that i can show a
    > >> warning to the user when further allocation is not possible.

    >
    > >> Same goes for hp ux 10 if somebody can help?
    > >> And also for suse linux 9 and redhat 9 ownwards?

    >
    > >> Isn't there a common api or a configuration file where i can retrieve
    > >> this information from, for most of the unixes?

    >
    > > On a modern operating system, there really isn't a good way to find out
    > > "how much more memory can I allocate before I run out". *The amount of
    > > memory available to your process depends on many factors, and is
    > > constantly changing based on what else is happening in the system.

    >
    > > Really, about the only thing you can reliably do is to try the
    > > allocation and see if it succeeds. *You can try to write the program in
    > > such a way that it will behave gracefully when this occurs, perhaps
    > > ask the user to free up some memory and then try to continue.

    >
    > That is usually the right approach, but AIUI it falls apart slightly on
    > systems which overcommit VM, like Linux. That is, they return memory to
    > the process even if there isn't enough VM. When you try to use the
    > returned memory... either you get lucky and the system's got enough VM
    > again or the OS has to kill some other poor process to make some VM for
    > you.
    >
    > I've a feeling HP-UX overcommits VM too.
    > --
    > Chris- Hide quoted text -
    >
    > - Show quoted text -


    Hi,

    So can't i get the threshold values for a process on each system about
    their Virtual address space available that would be of real benefit.
    Some numbers where I can tell user that you are now entering red zone,
    if i can't find the exact number.


    I am working on HP-UX 11i v1, suse linux 9, redhat linux 9 and solaris
    2.6.

    I now know thanks to Andrew Gabriel in the previous posts that in my
    sun4u i can have all the 4GB available to be allocatable but can't i
    get it through some api -- or can somebody point me to documentation
    for these systems mentioning the virtual memory available to the 32-
    bit process.

    Cheers,
    R




    Cheers,
    R

  13. Re: virtual address space size of a proces

    On Nov 5, 2:40*am, Rainer Weikusat wrote:

    > guidevelo...@gmail.com writes:
    > [...]


    > > I understand that the address space is 4 GB which could be used by
    > > kernel space or maybe completely used by user space, however i would
    > > like to know for solaris 2.4 ownwards, how can i programatically get
    > > the virtual memory available to the process so that i can show a
    > > warning to the user when further allocation is not possible.

    >
    > The answer is 'you cannot'.


    You can use getrlimit64 and specify that the user must set a sensible
    address space limit. Output an error message or warning about reduced
    functionality if this is not done.

    If running out of address space is a realistic possibility, and there
    are sensible things (like load shedding or map pruning) that you could
    do if you knew you were running low, then it is reasonable to demand
    this be set.

    DS

  14. Re: virtual address space size of a proces

    On Nov 4, 10:11*pm, Chris Ridd wrote:

    > That is usually the right approach, but AIUI it falls apart slightly on
    > systems which overcommit VM, like Linux. That is, they return memory to
    > the process even if there isn't enough VM. When you try to use the
    > returned memory... either you get lucky and the system's got enough VM
    > again or the OS has to kill some other poor process to make some VM for
    > you.
    >
    > I've a feeling HP-UX overcommits VM too.


    Unless the system is badly misconfigured (which isn't your fault) it
    should only overcommit vm such that an actual shortage is very, very
    rare. So rare that failing horribly in this scenario is as expected as
    failing horribly if bullets riddle the server.

    Setting overcommitment is saying "I am willing to have my programs
    fail if I somehow manage to use more than the amount of vm I actually
    have". Respecting the admin's wishes is not falling apart. It's doing
    the right thing.

    DS

  15. Re: virtual address space size of a proces

    Chris Ridd writes:
    > On 2008-11-05 10:39:03 +0000, Rainer Weikusat said:
    >> Chris Ridd writes:
    >>> On 2008-11-05 00:58:48 +0000, Nate Eldredge said:

    >> [...]
    >>
    >>>> Really, about the only thing you can reliably do is to try the
    >>>> allocation and see if it succeeds.

    >> [...]
    >>
    >>> That is usually the right approach, but AIUI it falls apart slightly
    >>> on systems which overcommit VM, like Linux. That is, they return
    >>> memory to the process even if there isn't enough VM. When you try to
    >>> use the returned memory... either you get lucky and the system's got
    >>> enough VM again or the OS has to kill some other poor process to make
    >>> some VM for you.

    >> The default is to kill the process which tried the memory access that
    >> could not succeed because there wasn't enough virtual memory
    >> available. But any other behaviour would not 'magically' cause this
    >> set of processes whose attempts to use the ressources of the system in
    >> particular way at that time to be more successful.

    >
    > I had in mind the infamous Linux "OOM killer" algorithm, which *will*
    > AIUI kill other things.


    If enabled, yes. Which happens to be the default for 2.6, while the
    other behaviour was the default for 2.4 (the old default policy can be
    restored via /proc/sys/vm/oom_kill_allocating_task).


  16. Re: virtual address space size of a proces

    On 5 Nov, 10:40, Rainer Weikusat wrote:
    > guidevelo...@gmail.com writes:
    >
    > [...]
    >
    > > I understand that the address space is 4 GB which could be used by
    > > kernel space or maybe completely used by user space, however i would
    > > like to know for solaris 2.4 ownwards, how can i programatically get
    > > the virtual memory available to the process so that i can show a
    > > warning to the user when further allocation is not possible.

    >
    > The answer is 'you cannot'.


    Hi,

    Fine i cannot find it directly but reading the link below it says that
    you can get the KERNELBASE to get the idea of the remaining virtual
    memory for user addressing for a process.
    http://docs.sun.com/app/docs/doc/817...dress+space%22


    I would like to know will this technique work or not?

    How can i know that how much do i have for kernel space and how much
    for user space in a process

    I am using sun4u SunOS 2.6.

    Also if somebody can help me out with HP-UX 11, suse and redhat linux
    9 regarding this.

    Cheers,
    R

  17. Re: virtual address space size of a proces

    guideveloper@gmail.com writes:
    > On 5 Nov, 10:40, Rainer Weikusat wrote:
    >> guidevelo...@gmail.com writes:
    >>
    >> [...]
    >>
    >> > I understand that the address space is 4 GB which could be used by
    >> > kernel space or maybe completely used by user space, however i would
    >> > like to know for solaris 2.4 ownwards, how can i programatically get
    >> > the virtual memory available to the process so that i can show a
    >> > warning to the user when further allocation is not possible.

    >>
    >> The answer is 'you cannot'.

    >
    > Hi,
    >
    > Fine i cannot find it directly but reading the link below it says that
    > you can get the KERNELBASE to get the idea of the remaining virtual
    > memory for user addressing for a process.


    That refers to the size of the address space for a process, not to
    the amount of virtual memory it can or cannot allocate.

  18. Re: virtual address space size of a proces

    On Tue, 4 Nov 2008, guideveloper@gmail.com wrote:

    > I understand that the address space is 4 GB which could be used by
    > kernel space or maybe completely used by user space, however i would
    > like to know for solaris 2.4 ownwards, how can i programatically get
    > the virtual memory available to the process so that i can show a
    > warning to the user when further allocation is not possible.


    You're thinking about this in the wrong way. WHat you should do is
    allocate memory as and when required, and report allocation failures
    to the user.

    The amount of virtual memory available to a process--within the limits
    others have defined--is dynamic. For example, how much swap space is
    available when you perform the check might change when you go to allocate
    memory, so you must check for failure anyway.

    It's a bit like using access() to check a file's permissions before
    opeing it. It's a pointless exercise because the perms could change
    between the access() and the open(), so you have to check the value
    from open() anyway. And given that, there's no point in calling
    access() in the first place!

    --
    Rich Teer, SCSA, SCNA, SCSECA

    CEO,
    My Online Home Inventory

    URLs: http://www.rite-group.com/rich
    http://www.linkedin.com/in/richteer
    http://www.myonlinehomeinventory.com

  19. Re: virtual address space size of a proces

    guideveloper@gmail.com writes:

    >I understand that the address space is 4 GB which could be used by
    >kernel space or maybe completely used by user space, however i would
    >like to know for solaris 2.4 ownwards, how can i programatically get
    >the virtual memory available to the process so that i can show a
    >warning to the user when further allocation is not possible.


    There are many reasons why allocation will fail (not enough swap,
    mmaping something where the heap wants to grow to, etc)

    The only way to find out is to test; and in that case *alloc() will
    return NULL.

    >Isn't there a common api or a configuration file where i can retrieve
    >this information from, for most of the unixes?


    Generally, this is going to be very difficuot and it's also not
    constant during the life of the process.

    What do you want to do when the memory runs out?

    Casper
    --
    Expressed in this posting are my opinions. They are in no way related
    to opinions held by my employer, Sun Microsystems.
    Statements on Sun products included here are not gospel and may
    be fiction rather than truth.

  20. Re: virtual address space size of a proces

    >I am looking to work on unix systems. I want to get the total virtual
    >address space of a process, the used virtual memory of a process i am
    >able to get without any problem. I have tried using getrlimit but that
    >gives only RLIMIT_INIFINITY or RLIMIT64_INFINITY if i use the
    >getrlimit64.
    >
    >Can you give me an idea on how much virtual addressable space is there
    >in each respective unix system for a user address space as i need to
    >do my memory allocations based on that and show a warning in an
    >application when the memory is about to get depleted?


    UNIX is not a uni-tasking system. It is going to be very difficult
    to make several processes based on the GREED memory allocation
    scheme (allocate everything I can before I know I'll need it)
    cooperate with each other if they don't know how. While it might
    be possible to make two instances of the same program cooperate
    with each other, it's not possible to do that with programs using
    the same strategy that don't know about each other.

    How likely is it that someone will want to dedicate an entire machine
    to running your program? If it's a database I can see the point,
    but usually for a database there are several dozen different things
    that can be tuned for how much memory to use, and correctly allocating
    the memory to the right pool is more important than grabbing all
    the memory you can. You're stuck with manual tuning from some kind
    of configuration file or command line arguments. The admin might
    want to allow enough space for backup programs to run (there *IS*
    data other than the database that needs backing up) without shutting
    down your application (it might be much easier to just shut down
    processing of queries than terminating and restarting the process).

    There are several limits to the virtual memory that can be allocated
    by a user process. You're stuck with the lowest one:

    1. Limits on the address space: 2**number_of_bits_in_a_pointer.
    This may be lower if there are bits for privilege levels or other
    such things in a pointer. That's 4GB for a 32-bit system. This is
    pretty much fixed for a given system.

    2. The kernel may take a chunk out of the address space in #1 for
    itself. You might be limited to 2GB or 3GB on a 32-bit system.
    This is pretty much fixed for a given system.

    3. Limits on backing store: The size of RAM + the size of swap/page
    space minus what everyone (including the process itself) has
    allocated. It's possible the admin could add swap space to a running
    system, but it's an unusual procedure. The amount of RAM and
    swap/page space is individual to every system, although there are
    guidelines. The totals of what everyone has allocated is *extremely*
    volatile as processes are created, allocate memory, and terminate.

    4. Administrative limits on virtual memory: you could probably
    see these with getrlimit(). The limit might be changable somewhat
    with setrlimit(), especially downward. It won't normally change
    without the program knowing about it during a given run.

    5. Policies to avoid deadlock might prevent a process from allocating
    more than a certain percentage of RAM + swap/page space.

    6. The amount of memory you can allocate without non-zero risk of
    the Linux OOM killer killing your process or some process that you
    require to function (e.g. syslog, nameserver, NFS daemon, mountd,
    etc.) is about what you've already got plus ZERO. No, that doesn't
    actually reduce the risk to zero, especially if there's another process
    around using the GREED memory allocation strategy.


    There is a good possibility that the GREED memory allocation strategy,
    especially if it's very accurately tuned, will have the process
    getting in its own way. For example, it might accurately grab
    exactly all the memory it can get, but not take into account that
    the system library needs memory for something the program will use
    later: stdio file buffers, name resolution, mapping in another
    shared library on the fly, etc. There's no point in warning the
    user to reduce memory at this point: if the message can even be
    sent to the user without running out of memory first, there may be
    nothing for the user to kill off that lets the process continue.

    There's also the possibility that allocating memory before you
    determine that you need it kills performance because of an increased
    amount of swapping. It might actually finish. Slowly.


+ Reply to Thread
Page 1 of 2 1 2 LastLast