Initialization of very large blocks of memory - HP UX

This is a discussion on Initialization of very large blocks of memory - HP UX ; HP-UX B.11.23 U ia64 Hi, We allocate GAB using valloc(): void* p1 = valloc ( ) After that, we initialize that memory: memset (p1, 0, ). Is there any way to manage performance ot the initializing (not necessarily using memset)? ...

+ Reply to Thread
Results 1 to 5 of 5

Thread: Initialization of very large blocks of memory

  1. Initialization of very large blocks of memory

    HP-UX B.11.23 U ia64

    Hi,

    We allocate GAB using valloc():
    void* p1 = valloc (<1GB>)

    After that, we initialize that memory:
    memset (p1, 0, <1GB>).

    Is there any way to manage performance ot the initializing (not necessarily using memset)?

    The same question concerning shared memory:
    int shmid = shmget (key, <1GB>, 0);
    char* p2 = shmat (shmid, 0, 0).
    memset (p2, 0, <1GB>).

    Thanks,
    Alex Vinokur



  2. Re: Initialization of very large blocks of memory


    "Alex Vinokur" wrote in message news:48fc5c5f$0$29392$834e42db@reader.greatnowhere .com...

    [snip]
    --------------------------------------
    > We allocate GAB using valloc():

    Must be:
    We allocate 1GB using valloc():
    Sorry
    -------------------------------------
    > void* p1 = valloc (<1GB>)


    [snip]

    Alex Vinokur



  3. Re: Initialization of very large blocks of memory

    Alex Vinokur wrote:
    > HP-UX B.11.23 U ia64
    >
    > Hi,
    >
    > We allocate GAB using valloc():
    > void* p1 = valloc (<1GB>)
    >
    > After that, we initialize that memory:
    > memset (p1, 0, <1GB>).
    >
    > Is there any way to manage performance ot the initializing (not necessarily using memset)?


    Only initialize the parts you're really touching? (Kind of depends
    on your algorithm... if you know what parts have been written to
    by whatever means [direct write, I/O, etc.] you know what parts are
    valid and what parts aren't).

    Since valloc() explicitly says it doesn't initialize and is managed by
    libc [which means I would presume this could be reuse of prior heap
    frees], I can't see how you'd be safe any other way.

    Personally, I'd just use MAP_PRIVATE|MAP_ANONYMOUS with mmap(2) for
    such an object unless there's a really pressing reason to let libc
    manage it. Now, you probably didn't do that because you can't
    request the 1Gb alignment here -- but that can be tweaked if you
    have admin privileges (or this is a very performance sensitive
    application). Private/Anonymous mmap uses the same page size hint
    (man vps_pagesize, chatr_ia(1) ) that the heap does [since they're
    both process data]. Which means that if you do a chatr +pd 1G your
    1Gb mmap() will try to get a 1Gb page... and hence will be 1Gb
    aligned. (if you go higher to 4Gb it won't matter.. there's a minimum
    between the hint and the size of the object, obviously). Alternately,
    you could just raise the default hint maximum for the whole system
    (vps_ceiling) -- but just chatr'ing the binary in question is simpler.

    The down side to that would be if you have Gb worth of heap as well
    but very sparse access (so multi-Gb of VSS but only kb or Mb of
    resident memory). The chatr hint would cause a fault in the heap
    to bring in a larger physical page.

    More in a moment....

    >
    > The same question concerning shared memory:
    > int shmid = shmget (key, <1GB>, 0);
    > char* p2 = shmat (shmid, 0, 0).
    > memset (p2, 0, <1GB>).
    >


    This one's much easier. Just don't memset() any shared object which
    has just been created. man pagezero_daemon_enabled(5) [about one of
    the only places I can find this written down], but the short of it
    is that the kernel will always pre-zero out memory it gives to user
    space for security anyway. So any fresh object (such as a new
    shmget+shmat pair) will have 0's on the first access from your
    application's point of view. The pagezero daemon is there to offload
    the expense of keeping such large blobs of memory zero'd out to your
    idle cycles. (If the daemon hasn't been able to run or demand is just
    outpacing it, the zero'ing is done in the context of the faulting
    thread within the kernel... so your memset() is just extra work here).

    Making your large private objects yourself via anonymous mmap() gives
    you the same confidence that the kernel regards this as new user pages
    and hence you'll know that the pages come pre-zeroed the same way. Hence
    my recommendation.

    (For anyone on prior systems reading this, 11.11 didn't have the
    daemon -- but the kernel promise still holds.)

    Don
    --
    kernel, n:
    A part of an operating system that preserves the medieval traditions
    of sorcery and black art.

  4. Re: Initialization of very large blocks of memory

    Alex Vinokur wrote:
    > Is there any way to manage performance of the initializing (not necessarily using memset)?



    Other than not doing it in the cases Don mentioned, there probably isn't
    any way faster than memset(3). It should be storing 16 bytes of zeros
    at a time.

  5. Re: Initialization of very large blocks of memory


    "Don Morris" wrote in message news:gdi6rj$it2$1@usenet01.boi.hp.com...
    > Alex Vinokur wrote:
    > > HP-UX B.11.23 U ia64
    > >
    > > Hi,
    > >
    > > We allocate GAB using valloc():
    > > void* p1 = valloc (<1GB>)
    > >
    > > After that, we initialize that memory:
    > > memset (p1, 0, <1GB>).
    > >
    > > Is there any way to manage performance ot the initializing (not necessarily using memset)?

    >
    > Only initialize the parts you're really touching? (Kind of depends
    > on your algorithm... if you know what parts have been written to
    > by whatever means [direct write, I/O, etc.] you know what parts are
    > valid and what parts aren't).
    >

    [snip]
    >
    > (For anyone on prior systems reading this, 11.11 didn't have the
    > daemon -- but the kernel promise still holds.)
    >


    Very useful and helpful information.
    Don, thank you.

    Alex Vinokur



+ Reply to Thread