Using 'xvfb' to simulate X-based program interaction - Xwindows

This is a discussion on Using 'xvfb' to simulate X-based program interaction - Xwindows ; All, I'm familiar with 'X' from a program use perspective (nowdays I usually use X-server-on-Cygwin-on-Windows and connect to RedHat Fedora Core 5 (FC5) with "ssh -X" or "ssh -Y" to use GUI programs). *** I want to simulate user interaction ...

+ Reply to Thread
Results 1 to 2 of 2

Thread: Using 'xvfb' to simulate X-based program interaction

  1. Using 'xvfb' to simulate X-based program interaction

    All,

    I'm familiar with 'X' from a program use perspective (nowdays I usually
    use
    X-server-on-Cygwin-on-Windows and connect to RedHat Fedora Core 5
    (FC5) with "ssh -X" or "ssh -Y" to use GUI programs).

    *** I want to simulate user interaction with X programs, though. ***
    I'm sure others have done this, I've looked online and can't find ready
    references to how to do this. I'm not familiar with X/Xlib programming
    and am not too sure on which "paradigm" to use.

    My goals:

    * run one or more X programs in controlled environment
    (e.g., Thunderbird e-mail client or Firefox web browser)
    * use a separate program to interact with the X programs,
    simulating mouse cursor movement, mouse clicks,
    keyboard input, and capturing screen elements
    (typically rectangular sub-portions of the entire root
    window or perhaps subportions of specific applications)

    Fundamental setup:

    * FUTURE_AND_EASY_ENOUGH: run daemon/service at startup to
    supervise
    entire operation
    * CURRENT_BY_HAND: run 'Xvfb :n' to start virtual X server
    listening on port 600n, DISPLAY="localhost:N.0"
    * CURRENT_BY_HAND: run typical X client program,
    e.g., "my_typical_X_client_GUI_program -display localhost:N.0"
    * CURRENT_BY_HAND: to verify setup working, I'm able to
    use 'xwd' program to dump X server root window and view with
    'gimp' or other image-display program
    * (ALTERNATIVE: I'm able to use 'Xnest' instead of 'Xvfb' to
    view start an interactive alternative X-server ...)
    * FUTURE_AND_DIFFICULT: ... OK, all the above works ...
    so here comes the part I'm not sure about ... *** what's the
    best way to simulate user interaction with the X programs
    and do screen captures?

    Simulating user interaction/screen capture, alternative approaches --
    which is best? which are possible?:

    * launch "master" non-GUI (no display) X client program that
    sends simulated mouse movement/button-click and keyboard events
    to the other X programs, and does screen captures
    * does this "master" X client need to be a "Window Manager",
    or can any X client fool with the other X clients?
    * does the "master" need to interact at the obscure "pixel"
    level, or can the "master" browse the window/widget hierarchy
    and find proper buttons/other-GUI-elements to send events to
    programmatically?
    * do I need to resort to C/C++ or are there Perl/Python/other
    libraries that provide a decent way to interact with
    X client programs in this manner?
    * do constant screen captures with 'xwd', write "low-level"
    mouse/keyboard drivers and hook them into the virtual
    X server somehow?
    * seems like a real bad alternative, a lot of work, need to
    resort to image-recognition, bla bla bla -- prohibitive
    and unrealistic

    CONCLUSION: Which approach is better? Are there references on
    how this is done? (I'm sure software testing suites already do
    a lot of this ... I don't want to shell out $$$ for my limited
    application.) Thanks.

    UrcaBraz


  2. Re: Using 'xvfb' to simulate X-based program interaction

    urcabraz@gmail.com wrote:
    > All,
    >
    > I'm familiar with 'X' from a program use perspective (nowdays I usually
    > use
    > X-server-on-Cygwin-on-Windows and connect to RedHat Fedora Core 5
    > (FC5) with "ssh -X" or "ssh -Y" to use GUI programs).
    >
    > *** I want to simulate user interaction with X programs, though. ***
    > I'm sure others have done this, I've looked online and can't find ready
    > references to how to do this. I'm not familiar with X/Xlib programming
    > and am not too sure on which "paradigm" to use.
    >
    > My goals:
    >
    > * run one or more X programs in controlled environment
    > (e.g., Thunderbird e-mail client or Firefox web browser)
    > * use a separate program to interact with the X programs,
    > simulating mouse cursor movement, mouse clicks,
    > keyboard input, and capturing screen elements
    > (typically rectangular sub-portions of the entire root
    > window or perhaps subportions of specific applications)
    >
    > Fundamental setup:
    >
    > * FUTURE_AND_EASY_ENOUGH: run daemon/service at startup to
    > supervise
    > entire operation
    > * CURRENT_BY_HAND: run 'Xvfb :n' to start virtual X server
    > listening on port 600n, DISPLAY="localhost:N.0"
    > * CURRENT_BY_HAND: run typical X client program,
    > e.g., "my_typical_X_client_GUI_program -display localhost:N.0"
    > * CURRENT_BY_HAND: to verify setup working, I'm able to
    > use 'xwd' program to dump X server root window and view with
    > 'gimp' or other image-display program
    > * (ALTERNATIVE: I'm able to use 'Xnest' instead of 'Xvfb' to
    > view start an interactive alternative X-server ...)
    > * FUTURE_AND_DIFFICULT: ... OK, all the above works ...
    > so here comes the part I'm not sure about ... *** what's the
    > best way to simulate user interaction with the X programs
    > and do screen captures?
    >
    > Simulating user interaction/screen capture, alternative approaches --
    > which is best? which are possible?:



    You can use XTest/XRecord to do what you want. They are server
    extensions that should be supported by Xvfb.

    Checkout my TkXext package that use those X extensions for recording
    events, and playing them back. It works with any X application.

    http://www.xmission.com/~georgeps/im...ftware/TkXext/

    I'm building an alternative to X11, and I plan to eventually modify Xvfb
    to work with my window system, similar to what you are planning. My end
    goal is to allow me to use applications like Firefox.


    > * launch "master" non-GUI (no display) X client program that
    > sends simulated mouse movement/button-click and keyboard events
    > to the other X programs, and does screen captures
    > * does this "master" X client need to be a "Window Manager",
    > or can any X client fool with the other X clients?
    > * does the "master" need to interact at the obscure "pixel"
    > level, or can the "master" browse the window/widget hierarchy
    > and find proper buttons/other-GUI-elements to send events to
    > programmatically?
    > * do I need to resort to C/C++ or are there Perl/Python/other
    > libraries that provide a decent way to interact with
    > X client programs in this manner?
    > * do constant screen captures with 'xwd', write "low-level"
    > mouse/keyboard drivers and hook them into the virtual
    > X server somehow?
    > * seems like a real bad alternative, a lot of work, need to
    > resort to image-recognition, bla bla bla -- prohibitive
    > and unrealistic


    This is where it gets iffy. You might want to hook into Xvfb somehow to
    trigger a callback after a framebuffer change. I'm not aware of any
    events generated specifically after a redraw that are consistent. I
    know that a GraphisExpose is sometimes generated, but they can be
    disabled.

    If you do manage to do this, please share your work, because I could use
    such code

    > CONCLUSION: Which approach is better? Are there references on
    > how this is done? (I'm sure software testing suites already do
    > a lot of this ... I don't want to shell out $$$ for my limited
    > application.) Thanks.


    Use my TkXext. It's BSD licensed

    The only other open source testing tool I'm aware of is Android, and
    it's GPLed.


    Regards,

    George

+ Reply to Thread