XIO: fatal IO error 11

[gdargaud]

At least, can you take a look at what calls this:

XGetImage (dpy=0x88a4b80, d=27263827, x=0, y=0, width=16, height=16, plane_mask=4294967295, format=2)

Is this call just in one place in cvi or in multiple places? Work from there... Possibly do a range checking and/or existence test before calling it.

[gdargaud]

OK, I finally got a crash in simulation mode on a simplish program. The source code is attached.

Compile and run it with:

$ unzip tp-test.zip && cd tp-test
$ cvicc -debug tp-tomographe && ulimit -c unlimited && ./tp-tomographe

Then simply click on [Start HV] and [Start]
It will generate a random 1800x180 2D plot (takes a few days, you can play with the timings in the ini file to try and speed it up, several instances can run at the same time).
If it doesn't crash till the end, simply press [Start] again. And Again.
The only crash I ever got in simulation mode took a week, but I got several crashes in real (i.e. hardware-connected) mode with much lower dimensions (50x180) so if there's a resource leak it's not correlated with the size of the 2D plot.

Note 1: because I simply used atexit(abort), a normal exit will also produce a core file which you can ignore.
Note 2: I've had this crash no matter what compiling options I use (-debug or not, -O1, -O3, -ggdb...)
Note 3: I'm begining to suspect some memory leak in CVI as all the core files I obtained so far (there are tens of them) are between 400 and 600Mb, while my apps use far less memory than that.


I can also provide you with my current exe and a 400Mb core file, but I'm not sure how useful that would be if you don't have the exact same setup. Upon starting gdb you'd get: warning: .dynamic section for "/usr/lib/something.so" is not at the expected address (wrong library or version mismatch?)

PS: I tried to upload a tgz file without success. You may or may not have to unzip with -a.

[gdargaud]

Bump...

I'm still waiting for a solution or at least an acknowledgement of this problem which I have on several apps on several systems. I'm surprised to be the only one.

[Kurt-P]

---

@gdargaud wrote:

I'm getting back to this critical problem with more info:

- it happens after days (or weeks) in intensive user interfaces (hundreds of updates every second)

- it happens with Scientific Linux 6.1 and 6.5 at least

- it happens in Mandriva 2010

- it happens with LabWindows for Linux 2010, 2013 and 2013p1 (version 13.0.0.29 and 13.0.0.30), although it happens more rarely in 2010

- I have it in several different programs compiled _from_ several different systems.

The culprit from the backtrace is always something internal to the CVI lib relating to Xwindows:

#21 0x00a706b1 in _XDefaultIOError () from /usr/lib/libX11.so.6
#22 0x00a70747 in _XIOError () from /usr/lib/libX11.so.6
#23 0x00a6f0a6 in _XReply () from /usr/lib/libX11.so.6
#24 0x00a53c0f in XGetImage () from /usr/lib/libX11.so.6

After much mostly fruitless research, my guess is that some X graphic property is used and not released but I'm not an X11 programmer...

Now this is a huge pita, because I may have uptime of a few years on my Linux control/command servers but it's useless if my processing programs crash after a few weeks.

I'd like this acknowledged as a serious bug by NI and I'm surprised that nobody else has been hit by this problem.

---

Hey gdargaud,

I saw your attachment in the previous post and was wondering which version of Scientific Linux you were using, and more importantly, which bitness (32 or 64). 

Thanks,

KP

[gdargaud]

Hello Kurt,

I've only ever used 32 bit versions of scientific Linux, never really troed to install CVI on 64-bit version as it doesn't work out of the box.

I've had this problem with 6.1 and I'm pretty sure with 6.4 and 6.5 as well (I downgraded, thinking it was running fine with previous versions, but that was wrong).

I haven't tried 7 as it doesn't seem to come in 32 bit version.

BTW: if you have more info on how to install CVI on 64-bit versions, I'm game.

[gdargaud]

Just so you know I managed to install CVI on Scientific Linux SL7 (64 bits), which was surprisingly easy. Last time I'd tried a 64-bit version there was a mess of dependencies. But after running for a week, the sample program I uploaded above crashed. I'd forgotten to enable "ulimit -c unlimited" by default so I didn't get a core to analyse and can't say yet if this is the same issue still present. We'll see in another week, sigh...

[Kurt-P]

Hey gdarguad, 

Glad to see you got it working on a 64-bit OS but for now I am going to focus on the 32-bit system. I am running the project right now on a virtual machine with Scientific Linux 6.6 32-bit and it has been going for about 5 days so far without any issue, at least no issue that I see. 

What kind of crash should I expect?

Is there a location on disk that you check for the core analysis files?

Is there any output in the command window when it crashes? 

I attached a picture of what the screen currently looks like, so let me know if there is a setting that is not correct. 

Thanks,

-KP

[gdargaud]

Hello Kurt and thanks for looking into it.

Looking at the screenshot, the prog has finished its pass. You should increase the values "Nb de Pas" and "Angle" to their max values and press [Start] again. And again, until it crashes. You can raise the max values in the .ini file just so it can run for a week without needing to restart: put MAX_NB_STEP = 2000 and MAX_NB_CYCLES = 720 and use those values in the interface.

As for what to expect, if you've done "ulimit -c unlimited" and compiled in debug mode "cvicc -debug tp-tomo.prj" and run the prog from the command line, you'll get a core file in the same directory and the message "XIO fatal IO error 11" on the console.

You can then simply analyse the core file with "gdb tp-tomographe core". Since the error is not actually in my code but in X11, you'll get warnings from gdb about missing source files, and suggestions on what to install (something like "sudo yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/ea/0dc9837abc2b4461df6f91a18fe78f5454a276"). Quit gdb, execute those and restart gdb and you'll be able to trace exactly the last 4 X11 functions before the crash (I gave them in a previous message). Above that are internal CVI calls to which I have no debugger access. Hopefully you will be able to gather more info from that. My hunch is that some graphical resource is not freed since the memory grows, but not mem to which my prog has access. I tried using valgrind and all kind of memory checkers without success.

Good luck !

[Kurt-P]

gdargaud,

Thank you for the clarification. I updated the ini file and adjusted the Nb de pas and Nb de rotations before I ran the file. This started yesterday, and it looks like it is going to run for about a week now. I'll let you know if I have any updates

I have a few general questions about the computer that these are running on. I looked through the post again and saw that it was running on 3 monitors with Nvidia drivers but did not find other hardware specs on the computer.

What are the other specs on the computer? For example, how many cores does the CPU have, how much RAM is on the computer, etc.

Have you tried to replicate this without the behavior with a different video driver?

Are these all actual machines that you've reproducing the issue on or have you replicated it on virtual machines as well?

I wanted to comment on 64-bit support for the CVI Linux Runtime Module. NI does support 32-bit binaries on supported 64-bit linux variants. We tested the module on RHEL 5,6; Scientific Linux 5,6; and OpenSuse 11.4-12.3. The issue that you were refering to (missing dependencies) I believe were due to a lack of 32-bit libraries on the machine I think the repositories that are enable by default are only 64-bit, but I could be wrong about this. Regardless, yum has a neat flag that allows you to find what package owns a particular shared object.

1. yum whatprovides  <shared object>

Then you can just use that to install the correct package. 

Thanks,

KP

[gdargaud]

Hello Kurt,

I just got two crashes in a virtual machine running the same soft I sent you. Took an entire week, running two instances simultaneously (beware of the log files taking up space). Here's the backtrace, similar to previously:

exit() handlers:
#0  0xf7737430 in __kernel_vsyscall ()
#1  0xf6e28656 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#2  0xf6e29e93 in __GI_abort () at abort.c:90
#3  0x08059bf5 in MyAbort ()
#4  0xf6e2bac9 in __run_exit_handlers (status=status@entry=1, 
    listp=0xf6fb43c4 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true)
    at exit.c:77
#5  0xf6e2bb2d in __GI_exit (status=status@entry=1) at exit.c:99

X11 code:
#6  0xf6cf1f0f in _XDefaultIOError (dpy=0x9768b80) at XlibInt.c:1292
#7  0xf6cf2154 in _XIOError (dpy=dpy@entry=0x9768b80) at XlibInt.c:1498
#8  0xf6cf0074 in _XReply (dpy=dpy@entry=0x9768b80, rep=rep@entry=0xff9f76fc, 
    extra=extra@entry=0, discard=discard@entry=1) at xcb_io.c:708
#9  0xf6cd392f in XGetGeometry (dpy=0x9768b80, d=88728260, root=0xff9f7868, 
    x=0xff9f7864, y=0xff9f7860, width=0xff9f7854, height=0xff9f7850, 
    borderWidth=0xff9f785c, depth=0xff9f7858) at GetGeom.c:47

CVI code:
#10 0xf759cccb in ?? () from /usr/local/lib/libcvi.so
#11 0xf7518d1d in ?? () from /usr/local/lib/libcvi.so
#12 0xf7519806 in ?? () from /usr/local/lib/libcvi.so
#13 0xf74f382f in ?? () from /usr/local/lib/libcvi.so
#14 0xf75e4929 in ?? () from /usr/local/lib/libcvi.so
#15 0xf75e4afa in ?? () from /usr/local/lib/libcvi.so
#16 0xf75e5208 in ?? () from /usr/local/lib/libcvi.so
#17 0xf75f3554 in ?? () from /usr/local/lib/libcvi.so
#18 0xf75f7dc3 in ?? () from /usr/local/lib/libcvi.so
#19 0xf74f01fe in ?? () from /usr/local/lib/libcvi.so
#20 0xf74ef2c4 in ?? () from /usr/local/lib/libcvi.so
#21 0xf74f170e in ?? () from /usr/local/lib/libcvi.so
#22 0xf7416538 in ?? () from /usr/local/lib/libcvi.so
#23 0xf7242fdb in ?? () from /usr/local/lib/libcvi.so

My code:
#24 0xf724453e in SetCtrlVal () from /usr/local/lib/libcvi.so
#25 0x08059784 in cb_SendSerial ()
#26 0x08052007 in SendSerial ()
#27 0x08052e56 in GetStatus ()
#28 0x0805a157 in UpdateStatus ()
#29 0x0805a0db in Wait ()
#30 0x0805c64b in cb_Start ()

RunUserInterface internals:
#31 0xf728bf7e in ?? () from /usr/local/lib/libcvi.so
#32 0xf728d9f9 in ?? () from /usr/local/lib/libcvi.so
#33 0xf729294b in ?? () from /usr/local/lib/libcvi.so
#34 0xf72e9057 in ?? () from /usr/local/lib/libcvi.so
#35 0xf7418f14 in ?? () from /usr/local/lib/libcvi.so
#36 0xf7562495 in ?? () from /usr/local/lib/libcvi.so
#37 0xf757c664 in ?? () from /usr/local/lib/libcvi.so
#38 0xf757ca9a in ?? () from /usr/local/lib/libcvi.so
#39 0xf7410432 in ?? () from /usr/local/lib/libcvi.so
#40 0xf7416fa6 in ?? () from /usr/local/lib/libcvi.so
#41 0xf755f564 in ?? () from /usr/local/lib/libcvi.so
#42 0xf756289b in ?? () from /usr/local/lib/libcvi.so
#43 0xf757c664 in ?? () from /usr/local/lib/libcvi.so
#44 0xf757ca9a in ?? () from /usr/local/lib/libcvi.so
#45 0xf757ceae in ?? () from /usr/local/lib/libcvi.so
#46 0xf76021af in ?? () from /usr/local/lib/libcvi.so
#47 0xf760371b in ?? () from /usr/local/lib/libcvi.so
#48 0xf7419084 in ?? () from /usr/local/lib/libcvi.so
#49 0xf728e1ca in ?? () from /usr/local/lib/libcvi.so
#50 0xf728e9ba in ?? () from /usr/local/lib/libcvi.so
#51 0xf728ed9d in ?? () from /usr/local/lib/libcvi.so

Top of the code:
#52 0xf728ee41 in RunUserInterface () from /usr/local/lib/libcvi.so
#53 0x0805076b in main ()

 As for your hardware questions, it's not relevant because I have this problem on all kind of systems, from recent 8-core to single core virtual machines. And any number of monitors. And all kind of SL from 6.1 to 7.0 (I don't think I ever had the problem with 5.x, but then I was running an older version of CVI for Linux).

Thanks for the 'whatprovides' tip, I didn't know that (I usually use Ubuntu, except for production systems where I use SL).