[2017-05-03 20:18:01.619691] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.11 (args: /usr/sbin/glusterfs --volfile-server=srvpve2g --volfi
le-id=datastore2 /mnt/pve/datastore2)
[2017-05-03 20:18:01.625747] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-03 20:18:01.625785] E [socket.c:2309:socket_connect_finish] 0-glusterfs: connection to 192.168.102.2:24007 failed (Connection refused)
[2017-05-03 20:18:01.625807] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: srvpve2g (Transport endpoint is not connected)
[2017-05-03 20:18:01.625816] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2017-05-03 20:18:01.625976] W [glusterfsd.c:1327:cleanup_and_exit] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xda) [0x7fe21a4bf59a] -->/usr/sbin/glusterfs(+0x1013e) [0x55d7ddf8a13e] -->/u
sr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d7ddf83967] ) 0-: received signum (1), shutting down
[2017-05-03 20:18:01.626002] I [fuse-bridge.c:5788:fini] 0-fuse: Unmounting '/mnt/pve/datastore2'.
[2017-05-04 13:03:21.997503] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.11 (args: /usr/sbin/glusterfs --volfile-server=srvpve3g --volfi
le-id=datastore2 /mnt/pve/datastore2)
[2017-05-04 13:03:22.005099] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-04 13:03:22.014008] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2017-05-04 13:03:22.014584] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-0: parent translators are ready, attempting connect on transport
[2017-05-04 13:03:22.014896] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-1: parent translators are ready, attempting connect on transport
[2017-05-04 13:03:22.015079] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-0: changing port to 49157 (from 0)
[2017-05-04 13:03:22.015112] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-2: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume datastore2-client-0
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host srvpve2g
  5:     option remote-subvolume /data/brick2/brick
  6:     option transport-type socket
  7:     option transport.address-family inet
  8:     option send-gids true
  9: end-volume
 10:  
 11: volume datastore2-client-1
 12:     type protocol/client
 13:     option ping-timeout 42
 14:     option remote-host srvpve3g
 15:     option remote-subvolume /data/brick2/brick
 16:     option transport-type socket
 17:     option transport.address-family inet
 18:     option send-gids true
 19: end-volume
 20:  
 21: volume datastore2-client-2
 22:     type protocol/client
 23:     option ping-timeout 42
 24:     option remote-host srvpve1g
 25:     option remote-subvolume /data/brick2/brick
 26:     option transport-type socket
 27:     option transport.address-family inet
 28:     option send-gids true
 29: end-volume
 30:  
 31: volume datastore2-replicate-0
 32:     type cluster/replicate
 33:     option arbiter-count 1
 34:     subvolumes datastore2-client-0 datastore2-client-1 datastore2-client-2
 35: end-volume
 36:  
 37: volume datastore2-dht
 38:     type cluster/distribute
 39:     option lock-migration off
 40:     subvolumes datastore2-replicate-0
 41: end-volume
 42:  
 43: volume datastore2-write-behind
 44:     type performance/write-behind
 45:     subvolumes datastore2-dht
 46: end-volume
 47:  
 48: volume datastore2-read-ahead
 49:     type performance/read-ahead
 50:     subvolumes datastore2-write-behind
 51: end-volume
 52:  
 53: volume datastore2-readdir-ahead
 54:     type performance/readdir-ahead
 55:     subvolumes datastore2-read-ahead
 56: end-volume
 57:  
 58: volume datastore2-io-cache
 59:     type performance/io-cache
 60:     subvolumes datastore2-readdir-ahead
 61: end-volume
 62:  
 63: volume datastore2-quick-read
 64:     type performance/quick-read
 65:     subvolumes datastore2-io-cache
 66: end-volume
 67:  
 68: volume datastore2-open-behind
 69:     type performance/open-behind
 70:     subvolumes datastore2-quick-read
 71: end-volume
 72:  
 73: volume datastore2-md-cache
 74:     type performance/md-cache
 75:     subvolumes datastore2-open-behind
 76: end-volume
 77:  
 78: volume datastore2
 79:     type debug/io-stats
 80:     option log-level INFO
 81:     option latency-measurement off
 82:     option count-fop-hits off
 83:     subvolumes datastore2-md-cache
 84: end-volume
 85:  
 86: volume meta-autoload
 87:     type meta
 88:     subvolumes datastore2
 89: end-volume
 90:  
+------------------------------------------------------------------------------+
[2017-05-04 13:03:22.015602] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:03:22.015885] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-0: Connected to datastore2-client-0, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:03:22.015896] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:03:22.015928] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-datastore2-replicate-0: Subvolume 'datastore2-client-0' came back up; going online.
[2017-05-04 13:03:22.015975] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-0: Server lk version = 1
[2017-05-04 13:03:22.015989] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-2: changing port to 49155 (from 0)
[2017-05-04 13:03:22.016024] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-1: changing port to 49153 (from 0)
[2017-05-04 13:03:22.016585] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:03:22.016780] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:03:22.016905] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-2: Connected to datastore2-client-2, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:03:22.016914] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:03:22.017039] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-2: Server lk version = 1
[2017-05-04 13:03:22.018221] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-1: Connected to datastore2-client-1, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:03:22.018236] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:03:22.023226] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-1: Server lk version = 1
[2017-05-04 13:03:22.023340] I [fuse-bridge.c:4147:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23
[2017-05-04 13:03:22.023382] I [fuse-bridge.c:4832:fuse_graph_sync] 0-fuse: switched to graph 0
[2017-05-04 13:03:22.024323] I [MSGID: 108031] [afr-common.c:2157:afr_local_discovery_cbk] 0-datastore2-replicate-0: selecting local read_child datastore2-client-0
[2017-05-04 13:08:12.034040] W [socket.c:590:__socket_rwv] 0-glusterfs: readv on 192.168.102.3:24007 failed (No data available)
[2017-05-04 13:08:12.034079] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: srvpve3g (No data available)
[2017-05-04 13:08:12.034096] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2017-05-04 13:08:23.028587] E [socket.c:2309:socket_connect_finish] 0-glusterfs: connection to 192.168.102.3:24007 failed (Connection refused)
[2017-05-04 13:08:35.897005] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2017-05-04 13:10:09.183071] W [glusterfsd.c:1327:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f7ac2490064] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x563aeb3b9af5] -->/
usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x563aeb3b9967] ) 0-: received signum (15), shutting down
[2017-05-04 13:10:09.183098] I [fuse-bridge.c:5788:fini] 0-fuse: Unmounting '/mnt/pve/datastore2'.
[2017-05-04 13:10:09.220777] I [fuse-bridge.c:5080:fuse_thread_proc] 0-fuse: unmounting /mnt/pve/datastore2
[2017-05-04 13:15:40.464333] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.11 (args: /usr/sbin/glusterfs --volfile-server=srvpve3g --volfi
le-id=datastore2 /mnt/pve/datastore2)
[2017-05-04 13:15:40.471061] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-04 13:15:40.479485] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2017-05-04 13:15:40.480057] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-0: parent translators are ready, attempting connect on transport
[2017-05-04 13:15:40.480371] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-1: parent translators are ready, attempting connect on transport
[2017-05-04 13:15:40.480577] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-0: changing port to 49157 (from 0)
[2017-05-04 13:15:40.480639] I [MSGID: 114020] [client.c:2356:notify] 0-datastore2-client-2: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume datastore2-client-0
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host srvpve2g
  5:     option remote-subvolume /data/brick2/brick
  6:     option transport-type socket
  7:     option transport.address-family inet
  8:     option send-gids true
  9: end-volume
 10:  
 11: volume datastore2-client-1
 12:     type protocol/client
 13:     option ping-timeout 42
 14:     option remote-host srvpve3g
 15:     option remote-subvolume /data/brick2/brick
 16:     option transport-type socket
 17:     option transport.address-family inet
 18:     option send-gids true
 19: end-volume
 20:  
 21: volume datastore2-client-2
 22:     type protocol/client
 23:     option ping-timeout 42
 24:     option remote-host srvpve1g
 25:     option remote-subvolume /data/brick2/brick
 26:     option transport-type socket
 27:     option transport.address-family inet
 28:     option send-gids true
 29: end-volume
 30:  
 31: volume datastore2-replicate-0
 32:     type cluster/replicate
 33:     option arbiter-count 1
 34:     subvolumes datastore2-client-0 datastore2-client-1 datastore2-client-2
 35: end-volume
 36:  
 37: volume datastore2-dht
 38:     type cluster/distribute
 39:     option lock-migration off
 40:     subvolumes datastore2-replicate-0
 41: end-volume
 42:  
 43: volume datastore2-write-behind
 44:     type performance/write-behind
 45:     subvolumes datastore2-dht
 46: end-volume
 47:  
 48: volume datastore2-read-ahead
 49:     type performance/read-ahead
 50:     subvolumes datastore2-write-behind
 51: end-volume
 52:  
 53: volume datastore2-readdir-ahead
 54:     type performance/readdir-ahead
 55:     subvolumes datastore2-read-ahead
 56: end-volume
 57:  
 58: volume datastore2-io-cache
 59:     type performance/io-cache
 60:     subvolumes datastore2-readdir-ahead
 61: end-volume
 62:  
 63: volume datastore2-quick-read
 64:     type performance/quick-read
 65:     subvolumes datastore2-io-cache
 66: end-volume
 67:  
 68: volume datastore2-open-behind
 69:     type performance/open-behind
 70:     subvolumes datastore2-quick-read
 71: end-volume
 72:  
 73: volume datastore2-md-cache
 74:     type performance/md-cache
 75:     subvolumes datastore2-open-behind
 76: end-volume
 77:  
 78: volume datastore2
 79:     type debug/io-stats
 80:     option log-level INFO
 81:     option latency-measurement off
 82:     option count-fop-hits off
 83:     subvolumes datastore2-md-cache
 84: end-volume
 85:  
 86: volume meta-autoload
 87:     type meta
 88:     subvolumes datastore2
 89: end-volume
 90:  
+------------------------------------------------------------------------------+
[2017-05-04 13:15:40.481282] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:15:40.481481] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-2: changing port to 49155 (from 0)
[2017-05-04 13:15:40.481616] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-datastore2-client-1: changing port to 49153 (from 0)
[2017-05-04 13:15:40.481811] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-0: Connected to datastore2-client-0, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:15:40.481823] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:15:40.481870] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-datastore2-replicate-0: Subvolume 'datastore2-client-0' came back up; going online.
[2017-05-04 13:15:40.481928] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-0: Server lk version = 1
[2017-05-04 13:15:40.482039] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:15:40.482333] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-datastore2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-04 13:15:40.482402] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-2: Connected to datastore2-client-2, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:15:40.482411] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:15:40.482537] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-2: Server lk version = 1
[2017-05-04 13:15:40.483155] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-datastore2-client-1: Connected to datastore2-client-1, attached to remote volume '/data/brick2/brick'.
[2017-05-04 13:15:40.483162] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-datastore2-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-04 13:15:40.487983] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-datastore2-client-1: Server lk version = 1
[2017-05-04 13:15:40.488107] I [fuse-bridge.c:4147:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23
[2017-05-04 13:15:40.488124] I [fuse-bridge.c:4832:fuse_graph_sync] 0-fuse: switched to graph 0
[2017-05-04 13:15:40.488930] I [MSGID: 108031] [afr-common.c:2157:afr_local_discovery_cbk] 0-datastore2-replicate-0: selecting local read_child datastore2-client-0