Slow vxWorks boot - shared memory related? - VxWorks

This is a discussion on Slow vxWorks boot - shared memory related? - VxWorks ; Hello all, We are running vxWorks 5.5.1 on two MVME5110 boards that interact across a VME backplane. We've used the vxWorks shared memory routines (VxMP) to implement much of our application. Being at the latter stages of our project's implementation ...

+ Reply to Thread
Results 1 to 14 of 14

Thread: Slow vxWorks boot - shared memory related?

  1. Slow vxWorks boot - shared memory related?

    Hello all,

    We are running vxWorks 5.5.1 on two MVME5110 boards that interact
    across a VME backplane. We've used the vxWorks shared memory routines
    (VxMP) to implement much of our application.

    Being at the latter stages of our project's implementation phase, there
    is one vxWorks-related bug which has become increasingly urgent for us
    to fix, which is this: after the vxWorks images are loaded, there is an
    unacceptably-long delay that occurs between then and when the
    application itself is able to run. (It's unacceptable in the sense that
    our application has to meet a start-up time constraint as a project
    requirement).

    Below is a sample log of our master board starting up, with a note
    added by me to show where the long delay (of about 40 seconds) occurs:

    ---------------------------------------------------
    VxWorks System Boot

    Copyright 1984-2002 Wind River Systems, Inc.

    CPU: Motorola MVME5110-2261 - MPC 7410
    Version: VxWorks5.5.1
    BSP version: 1.2/2
    Creation date: Aug 7 2005, 13:36:21

    Press any key to stop auto-boot...
    1 0
    auto-booting...

    boot device : tffs=0,0
    unit number : 0
    processor number : 0
    host name : HOST1
    file name : /tffs0/vxWorks
    inet on ethernet (e) : 192.168.0.1:ffffff00
    inet on backplane (b): 192.168.100.1:ffffff00
    host inet (h) : 192.168.0.100
    user (u) : vxworks
    ftp password (pw) : vxworks
    flags (f) : 0x8
    target name (tn) : host1
    other (o) : fei

    Attaching to TFFS... done.
    Loading /tffs0/vxWorks...1946032
    Starting at 0x100000...

    Attaching shared memory objects at 0x4100... done

    *****DELAY OCCURS HERE, APPROX 40 SECONDS******

    Attaching to SM net with memory anchor at 0x4100...
    smSetup:Warning! previous parameters differ!
    SM address: 192.168.100.1
    Attached TCP/IP interface to fei unit 0
    Attaching interface lo0...done
    Initializing backplane net with anchor at 0x4100... smSetup:Warning!
    previous parameters differ!
    done.
    Backplane anchor at 0x4100... Backplane IP address = 192.168.100.1
    Attaching network interface sm0... done.

    Adding 4696 symbols for standalone.

    ->
    ..... application loads normally
    ---------------------------------------------------

    We suspect that it is something that is related to the shared-memory
    network components, given that the delay occurs between the shared
    memory objects being attached and the connection to the SM network
    taking place. However, we've tinkered with including/excluding/altering
    several vxWorks components (shared memory and other) to no avail so
    far.

    Some other notes about our setup:
    - The problem occurs in an identical way whether the images are loaded
    from flash (TFFS) or from a host machine via FTP.
    - Excluding "auto addresss setup" or "default address for backplane"
    from the "shared memory network components" folder doesn't seem to aid
    the problem.

    I've read several threads about older vxWorks bugs that are related to
    blocked ports (when booting via FTP) causing a slow boot time, but I'm
    confident that this isn't the same problem. Could it be related to the
    way that our bootroms are configured? Any insights/questions/thoughts
    are very welcome and appreciated!

    Thanks kindly,
    Ben

    PS I would post our prjParams.h here but it's quite lengthy - if
    there's any specific queries you might have about our configuration,
    please feel free to ask.


  2. Re: Slow vxWorks boot - shared memory related?

    Ben wrote:
    > Hello all,
    >
    > We are running vxWorks 5.5.1 on two MVME5110 boards that interact
    > across a VME backplane. We've used the vxWorks shared memory routines
    > (VxMP) to implement much of our application.
    >
    > Being at the latter stages of our project's implementation phase, there
    > is one vxWorks-related bug which has become increasingly urgent for us
    > to fix, which is this: after the vxWorks images are loaded, there is an
    > unacceptably-long delay that occurs between then and when the
    > application itself is able to run. (It's unacceptable in the sense that
    > our application has to meet a start-up time constraint as a project
    > requirement).


    I have seen the same happen also, but it was on a system without shared
    memory. It was not a real problem for us, so we just left it.
    How much memory do you have? It might be related to memory
    initialization and/or MMU init or something like that.

    What you could do to check what is happening:
    - Check the ethernet ports to see if any traffic is coming out of your
    board.
    - See if you can add some debug info in the VxWorks startup, to see
    where the software is hanging.

    Kind regards,
    Johan

    --
    o o o o o o o . . . _____J_o_h_a_n___B_o_r_k_h_u_i_s___
    o _____ || http://www.borkhuis.com |
    .][__n_n_|DD[ ====_____ | johan@borkhuis.com |
    >(________|__|_[_________]_|________________________________|

    _/oo OOOOO oo` ooo ooo 'o!o!o o!o!o`
    == VxWorks FAQ: http://www.xs4all.nl/~borkhuis/vxworks/vxworks.html ==

  3. Re: Slow vxWorks boot - shared memory related?

    I would also check the VME bus traffic. Sounds like an event is pending
    and then it timeouts, hence the long delay.

    my $.05

    vxworks


  4. Re: Slow vxWorks boot - shared memory related?

    Thanks for the advice Johan and macabbi,

    I started out this morning looking at the network traffic that was
    emanating from the ethernet ports on our boards (and also our IPMC761
    I/O expansion module), and I didn't uncover anything particularly
    unusual.

    However, I have made some interesting findings by adding some trace
    statments to prjConfig.c. I've discovered that the delay occurs when
    usrNetworkInit () calls usrEndLibInit (). We are currently using END as
    a base for our shared memory network driver, in accordance with the
    WindRiver patch t22-cp1-smend - you can download this from WindSurf
    online support. When I remove END support from our kernel, the start-up
    is much, much quicker - however, it breaks our network and shared
    memory network configuration. I'm going to have a play around and see
    if I can revert to a BSD-based shared memory network.

    Being relatively new to VME debugging, I'm unsure how to go about
    "checking the VME bus traffic"... how would one go about doing this?

    Thanks again,
    Ben


  5. Re: Slow vxWorks boot - shared memory related?

    Hello Ben,
    Ben schrieb:
    > Being relatively new to VME debugging, I'm unsure how to go about
    > "checking the VME bus traffic"... how would one go about doing this?


    VMEbus is an asynchronous bus, which makes it happen that there is a bus
    controller.
    This bus controller looks at bus accesses.
    So when there is an access to a bus address, which does not exist, the
    bus controller sets up a time out (bus timeout), to abandon the access
    to the bus.
    This is done in hardware via a signal called bus error.
    If that occurs, there is a wrong address setting in the configuration of:
    the Shared Memory
    the VMEbus board, where you start the VxWorks on
    the VMEbus board, you want to get connected to
    So if the configured VMEboards are not plugged in or have not finished
    the initialisation of their shared memory software, the bus error may
    occur.
    The bus error can happen many times depending of the number of VMEbus
    accesses. So it may happen that it takes some seconds, where the VMEbus
    traffic prevents your bootstarp from finishing.

    Usually you can use a VMEbus tracing hardware, which is not that often
    in your shelf due to high costs.
    The other method is to take an osciloscope and watch the buserr line on
    the VMEbus.

    Hope it helps.

    --
    BaSystem Martin Raabe
    E: Martin.RaabeB-a-S-y-s-t-e-mde

  6. Re: Slow vxWorks boot - shared memory related?

    Ben wrote:
    > Thanks for the advice Johan and macabbi,
    >
    > I started out this morning looking at the network traffic that was
    > emanating from the ethernet ports on our boards (and also our IPMC761
    > I/O expansion module), and I didn't uncover anything particularly
    > unusual.
    >
    > However, I have made some interesting findings by adding some trace
    > statments to prjConfig.c. I've discovered that the delay occurs when
    > usrNetworkInit () calls usrEndLibInit (). We are currently using END as
    > a base for our shared memory network driver, in accordance with the
    > WindRiver patch t22-cp1-smend - you can download this from WindSurf
    > online support. When I remove END support from our kernel, the start-up
    > is much, much quicker - however, it breaks our network and shared
    > memory network configuration. I'm going to have a play around and see
    > if I can revert to a BSD-based shared memory network.


    You made some good progress here. What you could do to narrow it down
    more is look for the function usrEndLibInit and see where in this
    function the most time is spend. I don't know the exact location, but
    you can find this file somewhere under target/config IIRC.

    Kind regards,
    Johan

    --
    o o o o o o o . . . _____J_o_h_a_n___B_o_r_k_h_u_i_s___
    o _____ || http://www.borkhuis.com |
    .][__n_n_|DD[ ====_____ | johan@borkhuis.com |
    >(________|__|_[_________]_|________________________________|

    _/oo OOOOO oo` ooo ooo 'o!o!o o!o!o`
    == VxWorks FAQ: http://www.xs4all.nl/~borkhuis/vxworks/vxworks.html ==

  7. Re: Slow vxWorks boot - shared memory related?


    Martin Raabe wrote:
    > Hello Ben,
    > Ben schrieb:
    > > Being relatively new to VME debugging, I'm unsure how to go about
    > > "checking the VME bus traffic"... how would one go about doing this?

    >
    > VMEbus is an asynchronous bus, which makes it happen that there is a bus
    > controller.
    > This bus controller looks at bus accesses.
    > So when there is an access to a bus address, which does not exist, the
    > bus controller sets up a time out (bus timeout), to abandon the access
    > to the bus.
    > This is done in hardware via a signal called bus error.
    > If that occurs, there is a wrong address setting in the configuration of:
    > the Shared Memory
    > the VMEbus board, where you start the VxWorks on
    > the VMEbus board, you want to get connected to
    > So if the configured VMEboards are not plugged in or have not finished
    > the initialisation of their shared memory software, the bus error may
    > occur.
    > The bus error can happen many times depending of the number of VMEbus
    > accesses. So it may happen that it takes some seconds, where the VMEbus
    > traffic prevents your bootstarp from finishing.
    >
    > Usually you can use a VMEbus tracing hardware, which is not that often
    > in your shelf due to high costs.
    > The other method is to take an osciloscope and watch the buserr line on
    > the VMEbus.
    >
    > Hope it helps.
    >
    > --
    > BaSystem Martin Raabe
    > E: Martin.RaabeB-a-S-y-s-t-e-mde


    Very neat explaination Martin.Thanks for helping to understand.
    Regards,
    s.subbarayan


  8. Re: Slow vxWorks boot - shared memory related?

    Hi,
    on our system I had a very similar experience - its not VME however, but
    cPCI.
    In our case, it was due to no network being connected. If I removed the
    network plug (or if it was unterminated), the boot-sequence was considerably
    longer. If it was connected, the boot was quick..

    Don't know if this is relavant to you.

    Good luck anyways,
    -jostein


    "Ben" wrote in message
    news:1125379943.272645.91260@g44g2000cwa.googlegro ups.com...
    > Hello all,
    >
    > We are running vxWorks 5.5.1 on two MVME5110 boards that interact
    > across a VME backplane. We've used the vxWorks shared memory routines
    > (VxMP) to implement much of our application.
    >
    > Being at the latter stages of our project's implementation phase, there
    > is one vxWorks-related bug which has become increasingly urgent for us
    > to fix, which is this: after the vxWorks images are loaded, there is an
    > unacceptably-long delay that occurs between then and when the
    > application itself is able to run. (It's unacceptable in the sense that
    > our application has to meet a start-up time constraint as a project
    > requirement).
    >
    > Below is a sample log of our master board starting up, with a note
    > added by me to show where the long delay (of about 40 seconds) occurs:
    >
    > ---------------------------------------------------
    > VxWorks System Boot
    >
    > Copyright 1984-2002 Wind River Systems, Inc.
    >
    > CPU: Motorola MVME5110-2261 - MPC 7410
    > Version: VxWorks5.5.1
    > BSP version: 1.2/2
    > Creation date: Aug 7 2005, 13:36:21
    >
    > Press any key to stop auto-boot...
    > 1 0
    > auto-booting...
    >
    > boot device : tffs=0,0
    > unit number : 0
    > processor number : 0
    > host name : HOST1
    > file name : /tffs0/vxWorks
    > inet on ethernet (e) : 192.168.0.1:ffffff00
    > inet on backplane (b): 192.168.100.1:ffffff00
    > host inet (h) : 192.168.0.100
    > user (u) : vxworks
    > ftp password (pw) : vxworks
    > flags (f) : 0x8
    > target name (tn) : host1
    > other (o) : fei
    >
    > Attaching to TFFS... done.
    > Loading /tffs0/vxWorks...1946032
    > Starting at 0x100000...
    >
    > Attaching shared memory objects at 0x4100... done
    >
    > *****DELAY OCCURS HERE, APPROX 40 SECONDS******
    >
    > Attaching to SM net with memory anchor at 0x4100...
    > smSetup:Warning! previous parameters differ!
    > SM address: 192.168.100.1
    > Attached TCP/IP interface to fei unit 0
    > Attaching interface lo0...done
    > Initializing backplane net with anchor at 0x4100... smSetup:Warning!
    > previous parameters differ!
    > done.
    > Backplane anchor at 0x4100... Backplane IP address = 192.168.100.1
    > Attaching network interface sm0... done.
    >
    > Adding 4696 symbols for standalone.
    >
    > ->
    > .... application loads normally
    > ---------------------------------------------------
    >
    > We suspect that it is something that is related to the shared-memory
    > network components, given that the delay occurs between the shared
    > memory objects being attached and the connection to the SM network
    > taking place. However, we've tinkered with including/excluding/altering
    > several vxWorks components (shared memory and other) to no avail so
    > far.
    >
    > Some other notes about our setup:
    > - The problem occurs in an identical way whether the images are loaded
    > from flash (TFFS) or from a host machine via FTP.
    > - Excluding "auto addresss setup" or "default address for backplane"
    > from the "shared memory network components" folder doesn't seem to aid
    > the problem.
    >
    > I've read several threads about older vxWorks bugs that are related to
    > blocked ports (when booting via FTP) causing a slow boot time, but I'm
    > confident that this isn't the same problem. Could it be related to the
    > way that our bootroms are configured? Any insights/questions/thoughts
    > are very welcome and appreciated!
    >
    > Thanks kindly,
    > Ben
    >
    > PS I would post our prjParams.h here but it's quite lengthy - if
    > there's any specific queries you might have about our configuration,
    > please feel free to ask.
    >




  9. Re: Slow vxWorks boot - shared memory related?

    Thanks for the note on VME Martin - it had some good insights. I
    haven't actually had to go to an actual hardware level to investigate
    anything yet, but when I do that seems like a good tactic to start
    with.

    As far as my END problem goes, I think I have a reasonably clear idea
    of what's going on now. As part of usrEndLibInit, the MVME5100 BSP
    function sysFei82557EndLoad is called to initialise the END devices.
    This, in turn, calls fei82557EndLoad (pParamStr), which is contained in
    the vxWorks-provided driver for the END style 82557 Ethernet interface
    (target/src/drv/end/fei82557End.c).

    If I have a network cable plugged in to the board's Ethernet ports, and
    connected to something that is switched on (eg. my development PC), the
    driver initialises in about 2-3 seconds (I inserted some trace
    statements into sysFei82557EndLoad to check). However, if the Ethernet
    ports on the boards that are being initialised by the END driver are
    not plugged in to anything, then a delay of 15-20 seconds occurs. I'm
    guessing that somewhere in fei82557EndLoad, the driver tries to find an
    "active" physical connection (maybe at the data link layer, or
    something), although after a certain period it obviously times out and
    continues on regardless. While I can see the source code for the driver
    (in fei82558End.c), it seems to be pre-compiled into vxWorks so I'm
    unable to play around with it.

    As a workaround, I'm currently experimenting with disabling the
    Ethernet ports that we don't need to use (by removing them from the
    endDevTbl in our BSP's configNet.h), and simply ensuring that the
    Ethernet port we do want to use (which is located on our MVME761
    transition module, accessible via an IPMC761 PMC card) is plugged in to
    a "live" connection.

    We've also sent some queries off to our local WindRiver channels, so
    hopefully we may here something from them at some stage. Any further
    thoughts still appreciated! :-)

    Cheers,
    Ben


  10. Re: Slow vxWorks boot - shared memory related?

    Jostein - just saw your post straight after I made my last one. I think
    that this is, in fact, exactly the same problem that you're talking
    about. Thanks for that.

    Ben


  11. Re: Slow vxWorks boot - shared memory related?

    On 1 Sep 2005, ben.cheney@gmail.com wrote:

    > If I have a network cable plugged in to the board's Ethernet ports,
    > and connected to something that is switched on (eg. my development
    > PC), the driver initialises in about 2-3 seconds (I inserted some
    > trace statements into sysFei82557EndLoad to check). However, if the
    > Ethernet ports on the boards that are being initialised by the END
    > driver are not plugged in to anything, then a delay of 15-20 seconds
    > occurs. I'm guessing that somewhere in fei82557EndLoad, the driver


    Auto-negotiation. 10-base T or 10-base 2, 100Mbps or 10 Mbps. If it
    is possible, configure your network card (probably configNet.h) to
    match your hardware. This is often possible if the system design has
    the network configuration under your control. Obviously this won't
    work if you are making a generic network device (like a hub). Ergo,
    you must configure this.

    If your network has a heartbeat this will also speed the detection.
    Also, some hardware has a bit to detect connectivity. vxWorks does
    not have an interface to get this, but you can write your own code to
    access it.

    hth,
    Bill Pringlemeir.

    --
    A mind that is stretched to a new idea never returns to its original
    dimension - Anonymous

    vxWorks FAQ, "http://www.xs4all.nl/~borkhuis/vxworks/vxworks.html"

  12. Re: Slow vxWorks boot - shared memory related?

    > we're wondering if it's possible to make a kind of dummy Ethernet terminator?

    I'm not sure what I was thinking when I wrote that... nothing probably!
    We've mocked up a quick loopback RJ45 plug to do the job, seems to work
    fine.

    Ben


  13. Re: Slow vxWorks boot - shared memory related?

    Ben wrote:
    > My problem is that the highest level code which I can alter and
    > re-compile is at the BSP level (in config/mv5100/sysFei82557End.c), and
    > I don't have any options from there which are passed down to the driver
    > level to statically set connection speed/duplex settings etc. (Is there
    > a way to re-compile vxWorks drivers in /target/src/drv? I tried
    > altering them and rebuilding the vxWorks images/bootroms but the
    > changes don't seem to be included.)


    Try including the file src/drv/end/fei82557End.c into your project. This
    should rebuild the driver, and it will not include the procompiled
    driver version that is in the VxWorks library.

    Kind regards,
    Johan

    --
    o o o o o o o . . . _____J_o_h_a_n___B_o_r_k_h_u_i_s___
    o _____ || http://www.borkhuis.com |
    .][__n_n_|DD[ ====_____ | johan@borkhuis.com |
    >(________|__|_[_________]_|________________________________|

    _/oo OOOOO oo` ooo ooo 'o!o!o o!o!o`
    == VxWorks FAQ: http://www.xs4all.nl/~borkhuis/vxworks/vxworks.html ==

  14. Re: Slow vxWorks boot - shared memory related?

    Thanks for the tip Johan - that works a treat. I think I've now shaved
    off as much out of the vxWorks bootup time that I can, and I'm now
    trying to optimise our actual application.

    Thanks for the advice everyone - I've learned a lot!

    Ben


+ Reply to Thread