OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Re: daemon crash when shutting down large databases

From: Heikki Tuuri (Heikki.Tuuriinnodb.com)
Date: Sun Oct 02 2005 - 15:57:14 CDT


David,

man pthread_mutex_destroy says:

"
...
       pthread_mutex_destroy destroys a mutex object, freeing the
       resources it might hold. The mutex must be unlocked on
       entrance. In the LinuxThreads implementation, no resources
       are associated with mutex objects, thus
       pthread_mutex_destroy actually does nothing except check­
       ing that the mutex is unlocked.
...
       The pthread_mutex_destroy function returns the following
       error code on error:

              EBUSY the mutex is currently locked.
...
"

pthread mutexes are in InnoDB only created in a few places:

heikkihundin:~/mysql-5.0/innobase> grep -n os_fast_mutex_init ./*/*.c
./os/os0sync.c:137: os_fast_mutex_init(&(event->os_mutex));
./os/os0sync.c:453: os_fast_mutex_init(mutex);
./os/os0sync.c:562:os_fast_mutex_init(
./srv/srv0srv.c:930: os_fast_mutex_init(&srv_conc_mutex);
./srv/srv0start.c:1619: os_fast_mutex_init(&srv_os_test_mutex);
./sync/sync0sync.c:212: os_fast_mutex_init(&(mutex->os_fast_mutex));
./ut/ut0mem.c:51: os_fast_mutex_init(&ut_list_mutex);

In srv0start.c we clean up at a mysqld shutdown:

"
        /* 3. Free all InnoDB's own mutexes and the os_fast_mutexes inside
        them */
        sync_close();

        /* 4. Free the os_conc_mutex and all os_events and os_mutexes */

        srv_free();
        os_sync_free();

        /* Check that all read views are closed except read view owned
        by a purge. */

        if (UT_LIST_GET_LEN(trx_sys->view_list) > 1) {
                fprintf(stderr,
"InnoDB: Error: all read views were not closed before shutdown:\n"
"InnoDB: %lu read views open \n",
                UT_LIST_GET_LEN(trx_sys->view_list) - 1);
        }

        /* 5. Free all allocated memory and the os_fast_mutex created in
        ut0mem.c */

        ut_free_all_mem();
"

I think that, besides you, also someone else has reported the assertion
failure. I have suspected an OS bug but it might also be an InnoDB bug.
Maybe memory corruption.

Did you receive the exact same assertion failure also on x86?

How easy is it to repeat the crash?

I have now added some diagnostic code to InnoDB-4.1.15.

Regards,

Heikki

----- Original Message -----
From: "David Sparks" <davesactivestate.com>
Newsgroups: mailing.database.myodbc
Sent: Friday, September 23, 2005 9:03 PM
Subject: daemon crash when shutting down large databases

> mysql usually crashes when being shutdown. The machine is a dual AMD64
> w 8GB RAM running mysql-4.1.14 on Gentoo linux with a ~40GB database. I
> had similar crashes running 4.0.24 on an x86 running a ~275GB database.
>
> I always use `mysqladmin shutdown` rather than the init scripts to
> shutdown the daemon.
>
> Are there any known problems with shutting down large databases?
>
> Thanks,
>
> ds
>
> 050923 10:41:58 InnoDB: Starting shutdown...
> 050923 10:44:00InnoDB: Assertion failure in thread 1174235488 in file
> os0sync.c line 634
> InnoDB: Failing assertion: 0 == pthread_mutex_destroy(fast_mutex)
> InnoDB: We intentionally generate a memory trap.
> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
> InnoDB: If you get repeated assertion failures or crashes, even
> InnoDB: immediately after the mysqld startup, there may be
> InnoDB: corruption in the InnoDB tablespace. Please refer to
> InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html
> InnoDB: about forcing recovery.
> mysqld got signal 11;
> This could be because you hit a bug. It is also possible that this binary
> or one of the libraries it was linked against is corrupt, improperly
> built,
> or misconfigured. This error can also be caused by malfunctioning
> hardware.
> We will try our best to scrape up some info that will hopefully help
> diagnose
> the problem, but since we have already crashed, something is definitely
> wrong
> and this may fail.
>
> key_buffer_size=2147483648
> read_buffer_size=33550336
> max_used_connections=217
> max_connections=768
> threads_connected=0
> It is possible that mysqld could use up to
> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
> = 77591546 K
> bytes of memory
> Hope that's ok; if not, decrease some variables in the equation.
>
>
> --
> MySQL General Mailing List
> For list archives: http://lists.mysql.com/mysql
> To unsubscribe:
> http://lists.mysql.com/mysql?unsub=myodbcfreebsd.csie.nctu.edu.tw
>

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql