MYSQL SSD : c'est quoi ce bazard ?

Bon là ça commence à me gaver.

Le serveur SSD mis en prod pour mysql depuis 15 jours plante une fois par jour depuis maintenant une semaine et je sèche sur la cause


Mar 27 14:52:25 ns310016 mysqld: *** glibc detected *** /usr/sbin/mysqld: corrupted double-linked list: 0x00007f0d59cf2ff0 ***
Mar 27 14:52:25 ns310016 mysqld: ======= Backtrace: =========
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6[0x7f0d563d3928]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6[0x7f0d563d5868]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6(cfree+0x76)[0x7f0d563d5a36]
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(free_root+0x125)[0x7f0d5834a785]
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x263)[0x7f0d5808bf03]
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0xe8)[0x7f0d5808d508]
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(handle_one_connection+0x22e)[0x7f0d5807f8ae]
Mar 27 14:52:25 ns310016 mysqld: /lib/libpthread.so.0[0x7f0d578d3fc7]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6(clone+0x6d)[0x7f0d5642f59d]
Mar 27 14:52:25 ns310016 mysqld: ======= Memory map: ========
Mar 27 14:52:25 ns310016 mysqld: 402b2000-402b3000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 402b3000-402f3000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 403f0000-403f1000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 403f1000-40431000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 40610000-40611000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 40611000-40651000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4069f000-406a0000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 406a0000-406e0000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 40c09000-40c0a000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 40c0a000-40c4a000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 40c4a000-40c4b000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 40c4b000-40c8b000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 40cfb000-40cfc000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 40cfc000-40d3c000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 41002000-41003000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 41003000-41043000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4128c000-4128d000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4128d000-412cd000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 412cf000-412d0000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 412d0000-41310000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 413ce000-413cf000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 413cf000-4140f000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4152b000-4152c000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4152c000-4156c000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4176e000-4176f000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4176f000-417af000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4193e000-4193f000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4193f000-4197f000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 41986000-41987000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 41987000-419c7000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 419d2000-419d3000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 419d3000-41a13000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 41a13000-41a14000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 41a14000-41a54000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 41cbb000-41cbc000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 41cbc000-424bc000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 424bc000-424bd000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 424bd000-424fd000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 424fd000-424fe000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 424fe000-4253e000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4253e000-4253f000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 4253f000-4257f000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d00000000-7f0d02feb000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d02feb000-7f0d04000000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d08000000-7f0d0906a000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d0906a000-7f0d0c000000 ---p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d0d92b000-7f0d0d93b000 r-xp 00000000 09:01 117348 /lib/libresolv-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d0d93b000-7f0d0db3b000 ---p 00010000 09:01 117348 /lib/libresolv-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d0db3b000-7f0d0db3d000 rw-p 00010000 09:01 117348 /lib/libresolv-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d0db3d000-7f0d0db3f000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d0db3f000-7f0d0db43000 r-xp 00000000 09:01 117349 /lib/libnss_dns-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d0db43000-7f0d0dd42000 ---p 00004000 09:01 117349 /lib/libnss_dns-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d0dd42000-7f0d0dd44000 rw-p 00003000 09:01 117349 /lib/libnss_dns-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d0dd44000-7f0d55d41000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d55d41000-7f0d55d4b000 r-xp 00000000 09:01 115502 /lib/libnss_files-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d55d4b000-7f0d55f4b000 ---p 0000a000 09:01 115502 /lib/libnss_files-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d55f4b000-7f0d55f4d000 rw-p 0000a000 09:01 115502 /lib/libnss_files-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d55f4d000-7f0d55f56000 r-xp 00000000 09:01 115419 /lib/libnss_nis-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d55f56000-7f0d56156000 ---p 00009000 09:01 115419 /lib/libnss_nis-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d56156000-7f0d56158000 rw-p 00009000 09:01 115419 /lib/libnss_nis-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d56158000-7f0d5615f000 r-xp 00000000 09:01 117336 /lib/libnss_compat-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d5615f000-7f0d5635e000 ---p 00007000 09:01 117336 /lib/libnss_compat-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d5635e000-7f0d56360000 rw-p 00006000 09:01 117336 /lib/libnss_compat-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d56360000-7f0d564aa000 r-xp 00000000 09:01 117347 /lib/libc-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d564aa000-7f0d566a9000 ---p 0014a000 09:01 117347 /lib/libc-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d566a9000-7f0d566ac000 r--p 00149000 09:01 117347 /lib/libc-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d566ac000-7f0d566ae000 rw-p 0014c000 09:01 117347 /lib/libc-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d566ae000-7f0d566b3000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d566b3000-7f0d566c9000 r-xp 00000000 09:01 114977 /lib/libgcc_s.so.1
Mar 27 14:52:25 ns310016 mysqld: 7f0d566c9000-7f0d568c9000 ---p 00016000 09:01 114977 /lib/libgcc_s.so.1
Mar 27 14:52:25 ns310016 mysqld: 7f0d568c9000-7f0d568ca000 rw-p 00016000 09:01 114977 /lib/libgcc_s.so.1
Mar 27 14:52:25 ns310016 mysqld: 7f0d568ca000-7f0d5694c000 r-xp 00000000 09:01 117352 /lib/libm-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d5694c000-7f0d56b4b000 ---p 00082000 09:01 117352 /lib/libm-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d56b4b000-7f0d56b4d000 rw-p 00081000 09:01 117352 /lib/libm-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d56b4d000-7f0d56c3e000 r-xp 00000000 09:01 67299 /usr/lib/libstdc++.so.6.0.10
Mar 27 14:52:25 ns310016 mysqld: 7f0d56c3e000-7f0d56e3d000 ---p 000f1000 09:01 67299 /usr/lib/libstdc++.so.6.0.10
Mar 27 14:52:25 ns310016 mysqld: 7f0d56e3d000-7f0d56e43000 r--p 000f0000 09:01 67299 /usr/lib/libstdc++.so.6.0.10
Mar 27 14:52:25 ns310016 mysqld: 7f0d56e43000-7f0d56e46000 rw-p 000f6000 09:01 67299 /usr/lib/libstdc++.so.6.0.10
Mar 27 14:52:25 ns310016 mysqld: 7f0d56e46000-7f0d56e59000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d56e59000-7f0d56e6e000 r-xp 00000000 09:01 117346 /lib/libnsl-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d56e6e000-7f0d5706d000 ---p 00015000 09:01 117346 /lib/libnsl-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d5706d000-7f0d5706f000 rw-p 00014000 09:01 117346 /lib/libnsl-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d5706f000-7f0d57071000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d57071000-7f0d57079000 r-xp 00000000 09:01 117345 /lib/libcrypt-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d57079000-7f0d57279000 ---p 00008000 09:01 117345 /lib/libcrypt-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d57279000-7f0d5727b000 rw-p 00008000 09:01 117345 /lib/libcrypt-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d5727b000-7f0d572a9000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d572a9000-7f0d572ab000 r-xp 00000000 09:01 117339 /lib/libdl-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d572ab000-7f0d574ab000 ---p 00002000 09:01 117339 /lib/libdl-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d574ab000-7f0d574ad000 rw-p 00002000 09:01 117339 /lib/libdl-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d574ad000-7f0d574b5000 r-xp 00000000 09:01 115012 /lib/libwrap.so.0.7.6
Mar 27 14:52:25 ns310016 mysqld: 7f0d574b5000-7f0d576b4000 ---p 00008000 09:01 115012 /lib/libwrap.so.0.7.6
Mar 27 14:52:25 ns310016 mysqld: 7f0d576b4000-7f0d576b6000 rw-p 00007000 09:01 115012 /lib/libwrap.so.0.7.6
Mar 27 14:52:25 ns310016 mysqld: 7f0d576b6000-7f0d576cc000 r-xp 00000000 09:01 67306 /usr/lib/libz.so.1.2.3.3
Mar 27 14:52:25 ns310016 mysqld: 7f0d576cc000-7f0d578cc000 ---p 00016000 09:01 67306 /usr/lib/libz.so.1.2.3.3
Mar 27 14:52:25 ns310016 mysqld: 7f0d578cc000-7f0d578cd000 rw-p 00016000 09:01 67306 /usr/lib/libz.so.1.2.3.3
Mar 27 14:52:25 ns310016 mysqld: 7f0d578cd000-7f0d578e3000 r-xp 00000000 09:01 117350 /lib/libpthread-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d578e3000-7f0d57ae3000 ---p 00016000 09:01 117350 /lib/libpthread-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d57ae3000-7f0d57ae5000 rw-p 00016000 09:01 117350 /lib/libpthread-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d57ae5000-7f0d57ae9000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d57ae9000-7f0d57b05000 r-xp 00000000 09:01 117341 /lib/ld-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d57cf8000-7f0d57cfd000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d57d01000-7f0d57d04000 rw-p 00000000 00:00 0
Mar 27 14:52:25 ns310016 mysqld: 7f0d57d04000-7f0d57d06000 rw-p 0001b000 09:01 117341 /lib/ld-2.7.so
Mar 27 14:52:25 ns310016 mysqld: 7f0d57d06000-7f0d585bc000 r-xp 00000000 09:01 73973 /usr/sbin/mysqld
Mar 27 14:52:25 ns310016 mysqld: 7f0d587bb000-7f0d58824000 r--p 008b5000 09:01 73973 /usr/sbin/mysqld
Mar 27 14:52:25 ns310016 mysqld: 7f0d58824000-7f0d58897000 rw-p 0091e000 09:01 73973 /usr/sbin/mysqld
Mar 27 14:52:25 ns310016 mysqld: 7f0d58897000-7f0d5ac99000 rw-p 00000000 00:00 0 [heap]
Mar 27 14:52:25 ns310016 mysqld: 7fffddecb000-7fffddee0000 rw-p 00000000 00:00 0 [stack]
Mar 27 14:52:25 ns310016 mysqld: 7fffddfff000-7fffde000000 r-xp 00000000 00:00 0 [vdso]
Mar 27 14:52:25 ns310016 mysqld: ffffffffff600000-ffffffffff601000 r--p 00000000 00:00 0 [vsyscall]
Mar 27 14:52:25 ns310016 mysqld: 100327 14:52:25 - mysqld got signal 6 ;
Mar 27 14:52:25 ns310016 mysqld: This could be because you hit a bug. It is also possible that this binary
Mar 27 14:52:25 ns310016 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Mar 27 14:52:25 ns310016 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Mar 27 14:52:25 ns310016 mysqld: We will try our best to scrape up some info that will hopefully help diagnose
Mar 27 14:52:25 ns310016 mysqld: the problem, but since we have already crashed, something is definitely wrong
Mar 27 14:52:25 ns310016 mysqld: and this may fail.
Mar 27 14:52:25 ns310016 mysqld: key_buffer_size=1073741824
Mar 27 14:52:25 ns310016 mysqld: read_buffer_size=4194304
Mar 27 14:52:25 ns310016 mysqld: It is possible that mysqld could use up to
Mar 27 14:52:25 ns310016 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2596353 K
Mar 27 14:52:25 ns310016 mysqld: bytes of memory
Mar 27 14:52:25 ns310016 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Mar 27 14:52:25 ns310016 mysqld:
Mar 27 14:52:25 ns310016 mysqld: thd: 0x7f0d0004e9d0
Mar 27 14:52:25 ns310016 mysqld: Attempting backtrace. You can use the following information to find out
Mar 27 14:52:25 ns310016 mysqld: where mysqld died. If you see no messages after this, something went
Mar 27 14:52:25 ns310016 mysqld: terribly wrong...
Mar 27 14:52:25 ns310016 mysqld: stack_bottom = 0x402f2100 thread_stack 0x40000
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7f0d5835f4b9]
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(handle_segfault+0x36a) [0x7f0d58074dfa]
Mar 27 14:52:25 ns310016 mysqld: /lib/libpthread.so.0 [0x7f0d578dba80]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6(gsignal+0x35) [0x7f0d56391ed5]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6(abort+0x183) [0x7f0d563933f3]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6 [0x7f0d563ce388]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6 [0x7f0d563d3928]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6 [0x7f0d563d5868]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6(cfree+0x76) [0x7f0d563d5a36]
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(free_root+0x125) [0x7f0d5834a785]
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x263) [0x7f0d5808bf03]
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(do_command(THD*)+0xe8) [0x7f0d5808d508]
Mar 27 14:52:25 ns310016 mysqld: /usr/sbin/mysqld(handle_one_connection+0x22e) [0x7f0d5807f8ae]
Mar 27 14:52:25 ns310016 mysqld: /lib/libpthread.so.0 [0x7f0d578d3fc7]
Mar 27 14:52:25 ns310016 mysqld: /lib/libc.so.6(clone+0x6d) [0x7f0d5642f59d]
Mar 27 14:52:25 ns310016 mysqld: Trying to get some variables.
Mar 27 14:52:25 ns310016 mysqld: Some pointers may be invalid and cause the dump to abort...
Mar 27 14:52:25 ns310016 mysqld: thd->query at (nil) is an invalid pointer
Mar 27 14:52:25 ns310016 mysqld: thd->thread_id=294568
Mar 27 14:52:25 ns310016 mysqld: thd->killed=NOT_KILLED
Mar 27 14:52:25 ns310016 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Mar 27 14:52:25 ns310016 mysqld: information that should help you find out what is causing the crash.




Au début je pensais que c'était un soucis de conf, mais d'abord je vois pas pourquoi étant donné que j'utilise la même conf depuis plusieurs années et qu'elle marche très bien sur d'autres serveurs...

oui j'ai vu mais c'est ça que je trouve curieux...

effectivement il semble qu'une mise à jour s'est mal passée... car je viens de voir ce message dans phpmyadmin (que je devrais utiliser plus souvent)
La version de votre bibliothèque MySQL (5.0.51a) diffère de la version de votre serveur MySQL (5.1.44). Ceci peut occasionner un comportement imprévisible.

maintenant c'est strange quand même comment c'est-y possible...