Hi all,
you see below the my.cnf and extractions of the logs on the joiner and donor. Unfortunately, I can't find out the problem, so I do need your help. I started the first node with /etc/init.d/mysql bootstrap-pxc.
Thank you in advance.
Cemil
DONOR - my.cnf
[mysqld]
binlog_format = ROW
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
innodb_locks_unsafe_for_binlog = 1
innodb_log_file_size = 512M
innodb_data_file_path = ibdata1:24M:autoextend
innodb_buffer_pool_size = 12G
#innodb_flush_log_at_trx_commit = 2
query_cache_size = 0
query_cache_type = 0
bind-address = 0.0.0.0
log_slave_updates
##
## WSREP options
##
wsrep_provider = /services/mysql/lib/libgalera_smm.so
wsrep_provider_options = "gcache.size=256M"
wsrep_cluster_name = mysql_cluster
wsrep_cluster_address = gcomm://192.168.128.196,192.168.128.197,192.168.128.198
wsrep_node_name = db1-private
wsrep_node_address = 192.168.128.196
wsrep_sst_method = xtrabackup-v2
wsrep_sst_auth = puser:secret
# Optional wsrep Provider Settings
wsrep_node_incoming_address = 192.168.128.196
wsrep_sst_receive_address = 192.168.128.196
wsrep_slave_threads = 8
#wsrep_replicate_myisam = 1
JOINER - my.cnf
[mysqld]
binlog_format = ROW
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
innodb_locks_unsafe_for_binlog = 1
innodb_log_file_size = 512M
innodb_data_file_path = ibdata1:24M:autoextend
innodb_buffer_pool_size = 12G
#innodb_flush_log_at_trx_commit = 2
query_cache_size = 0
query_cache_type = 0
bind-address = 0.0.0.0
log_slave_updates
##
## WSREP options
##
wsrep_provider = /services/mysql/lib/libgalera_smm.so
wsrep_provider_options = "gcache.size=256M"
wsrep_cluster_name = mysql_cluster
wsrep_cluster_address = gcomm://192.168.128.196,192.168.128.197,192.168.128.198
wsrep_node_name = db2-private
wsrep_node_address = 192.168.128.197
wsrep_sst_method = xtrabackup-v2
wsrep_sst_auth = puser:secret
# Optional wsrep Provider Settings
wsrep_node_incoming_address = 192.168.128.197
wsrep_sst_receive_address = 192.168.128.197
wsrep_slave_threads = 8
#wsrep_replicate_myisam = 1
JOINER - ERROR-LOG
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20150526 16:15:27.N)
xbstream: Can't create/write to file '././backup-my.cnf' (Errcode: 2 - No such file or directory)
xbstream: failed to create file.
2015/05/26 16:15:38 socat[82119] E write(1, 0x802451000, 41): Broken pipe
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 1 (20150526 16:15:38.N)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20150526 16:15:38.N)
2015-05-26 16:15:38 81866 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.128.197' --auth 'puser:secret' --datadir '/services/mysql/data/' --defaults-file '/services/mysql/my.cnf' --parent '81866' '' : 32 (Broken pipe)
2015-05-26 16:15:38 81866 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-05-26 16:15:38 81866 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2015-05-26 16:15:38 81866 [ERROR] Aborting
2015-05-26 16:15:38 81866 [Warning] WSREP: 0.0 (db1-private): State transfer to 1.0 (db2-private) failed: -22 (Invalid argument)
2015-05-26 16:15:38 81866 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.
2015-05-26 16:15:38 81866 [Note] WSREP: gcomm: terminating thread
2015-05-26 16:15:38 81866 [Note] WSREP: gcomm: joining thread
2015-05-26 16:15:38 81866 [Note] WSREP: gcomm: closing backend
2015-05-26 16:15:39 81866 [Note] WSREP: view(view_id(NON_PRIM,6641ad6c,18) memb {
a60e3d2b,0
} joined {
} left {
} partitioned {
6641ad6c,0
})
2015-05-26 16:15:39 81866 [Note] WSREP: view((empty))
2015-05-26 16:15:39 81866 [Note] WSREP: gcomm: closed
2015-05-26 16:15:39 81866 [Note] WSREP: mysqld: Terminated.
....
DONOR - ERROR-LOG
....
WSREP_SST: [INFO] Streaming with xbstream (20150526 16:15:27.N)
WSREP_SST: [INFO] Using socat as streamer (20150526 16:15:27.N)
WSREP_SST: [INFO] Using /tmp/tmp.4O9pcm11 as xtrabackup temporary directory (20150526 16:15:27.N)
WSREP_SST: [INFO] Using /tmp/tmp.2zGAaXD6 as innobackupex temporary directory (20150526 16:15:27.N)
WSREP_SST: [INFO] Streaming GTID file before SST (20150526 16:15:27.N)
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio TCP:192.168.128.197:4444; RC=( ${PIPESTATUS[@]} ) (20150526 16:15:27.N)
WSREP_SST: [INFO] Sleeping before data transfer for SST (20150526 16:15:27.N)
WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.128.197 4444 (20150526 16:15:37.N)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/services/mysql/my.cnf --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2>${DATA}/innobackup.backup.log | socat
-u stdio TCP:192.168.128.197:4444; RC=( ${PIPESTATUS[@]} ) (20150526 16:15:37.N)
2015/05/26 16:15:38 socat[25151] E write(5, 0x802451000, 8192): Broken pipe
WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /services/mysql/data//innobackup.backup.log (20150526 16:15:38.N)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (20150526 16:15:38.N)
WSREP_SST: [INFO] Cleaning up temporary directories (20150526 16:15:38.N)
2015-05-26 16:15:38 57661 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.128.197:4444/xtrabackup_sst//1' --auth 'puser:secret' --socket '/tmp/mysql.
sock' --datadir '/services/mysql/data/' --defaults-file '/services/mysql/my.cnf' '' --gtid '8e7b1114-c27b-11e4-a13f-f6862ce824b1:4625349'
2015-05-26 16:15:38 57661 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.128.197:4444/xtrabackup_sst//1' --auth 'puser:secret' --socket '/t
mp/mysql.sock' --datadir '/services/mysql/data/' --defaults-file '/services/mysql/my.cnf' '' --gtid '8e7b1114-c27b-11e4-a13f-f6862ce824b1:4625349': 22 (Invalid argument)
2015-05-26 16:15:38 57661 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.128.197:4444/xtrabackup_sst//1' --auth 'puser:secret' --socket '/tmp/mysql.
sock' --datadir '/services/mysql/data/' --defaults-file '/services/mysql/my.cnf' '' --gtid '8e7b1114-c27b-11e4-a13f-f6862ce824b1:4625349'
2015-05-26 16:15:38 57661 [Warning] WSREP: 0.0 (db1-private): State transfer to 1.0 (db2-private) failed: -22 (Invalid argument)
...
DONOR - innobackup.backup.log
150526 16:15:37 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/services/mysql/my.cnf;mysql_read_default_group=xtrabackup;mysql_s ocket=/tmp/mysql.sock' as 'pu
ser' (using password: YES).
150526 16:15:37 innobackupex: Connected to MySQL server
150526 16:15:37 innobackupex: Starting the backup operation
innobackupex: Using server version 5.6.22-72.0
innobackupex: Created backup directory /tmp/tmp.2zGAaXD6
150526 16:15:37 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/services/mysql/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp/tmp.4O9pcm11
--tmpdir=/tmp/tmp.4O9pcm11 --extra-lsndir='/tmp/tmp.4O9pcm11' --stream=xbstream
innobackupex: Waiting for ibbackup (pid=25154) to suspend
innobackupex: Suspend file '/tmp/tmp.4O9pcm11/xtrabackup_suspended_2'
xtrabackup version 2.2.10 based on MySQL server 5.6.22 FreeBSD10.1 (amd64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /services/mysql/data
xtrabackup: open files limit requested 0, set to 942318
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:24M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 536870912
>> log scanned up to (58176722719)
xtrabackup: Generating a list of tablespaces
[01] Streaming ./ibdata1
^Gxtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
^Gxtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
innobackupex: got a fatal error with the following stacktrace: at /services/mysql/bin/innobackupex line 2703.
main::wait_for_ibbackup_file_create('/tmp/tmp.4O9pcm11/xtrabackup_suspended_2') called at /services/mysql/bin/innobackupex line 2723
main::wait_for_ibbackup_suspend('/tmp/tmp.4O9pcm11/xtrabackup_suspended_2') called at /services/mysql/bin/innobackupex line 1976
main::backup() called at /services/mysql/bin/innobackupex line 1601
innobackupex: Error: The xtrabackup child process has died at /services/mysql/bin/innobackupex line 2703.