Skip to content
This repository was archived by the owner on Oct 23, 2024. It is now read-only.

* Updated for ES 6.6.1 #7

Merged
merged 3 commits into from
Jun 21, 2019
Merged

* Updated for ES 6.6.1 #7

merged 3 commits into from
Jun 21, 2019

Conversation

mpereira
Copy link

This PR adds support for Elasticsearch 6.6.1.

The main changes required in the Java code were related to AbstractLifecycleComponent not extending AbstractComponent anymore, so its previous logging facilities are now unavailable. The changes in the Elasticsearch codebase now make classes instantiate their own logger objects, so that's the path I took.

See more details in elastic/elasticsearch#35560.

Clean maven build output:

$ mvn clean package -Djava.security.policy=src/test/resources/plugin-security-test.policy -Dtests.gradle=true
[INFO] Scanning for projects...
[INFO] 
[INFO] ----------------< com.automattic:elasticsearch-statsd >-----------------
[INFO] Building elasticsearch-statsd 6.6.1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ elasticsearch-statsd ---
[INFO] Deleting /Users/murilo/mesosphere/elasticsearch-statsd-plugin/target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ elasticsearch-statsd ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/murilo/mesosphere/elasticsearch-statsd-plugin/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ elasticsearch-statsd ---
[INFO] Compiling 7 source files to /Users/murilo/mesosphere/elasticsearch-statsd-plugin/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ elasticsearch-statsd ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ elasticsearch-statsd ---
[INFO] Compiling 3 source files to /Users/murilo/mesosphere/elasticsearch-statsd-plugin/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ elasticsearch-statsd ---
[INFO] Surefire report directory: /Users/murilo/mesosphere/elasticsearch-statsd-plugin/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.automattic.elasticsearch.statsd.test.StatsdPluginIntegrationTest
[2019-03-11T20:13:47,182][WARN ][o.e.b.JNANatives         ] [[SUITE-StatsdPluginIntegrationTest-seed#[D8C8958638594349]]] Unable to lock JVM Memory: error=78, reason=Function not implemented
[2019-03-11T20:13:47,185][WARN ][o.e.b.JNANatives         ] [[SUITE-StatsdPluginIntegrationTest-seed#[D8C8958638594349]]] This can result in part of the JVM being swapped out.
[2019-03-12T07:13:47,603][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [masterFailOverShouldWork] before test
[2019-03-12T07:13:47,607][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [masterFailOverShouldWork] [StatsdPluginIntegrationTest#masterFailOverShouldWork]: setting up test
[2019-03-12T07:13:47,630][INFO ][o.e.t.InternalTestCluster] [masterFailOverShouldWork] Setup InternalTestCluster [SUITE-CHILD_VM=[0]-CLUSTER_SEED=[-6415491860216096516]-HASH=[1D765E33602D6]-cluster] with seed [A6F79824D468D8FC] using [0] dedicated masters, [3] (data) nodes and [0] coord only nodes (min_master_nodes are [auto-managed])
[2019-03-12T07:13:48,044][INFO ][o.e.e.NodeEnvironment    ] [masterFailOverShouldWork] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [129.4gb], net total_space [465.6gb], types [apfs]
[2019-03-12T07:13:48,045][INFO ][o.e.e.NodeEnvironment    ] [masterFailOverShouldWork] heap size [3.5gb], compressed ordinary object pointers [true]
[2019-03-12T07:13:48,048][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] node name [node_s0], node ID [lmEpwuT6R0GleHXKBBYH6A]
[2019-03-12T07:13:48,049][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] version[6.6.1], pid[87986], build[unknown/unknown/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Mac OS X/10.14.3/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_152/25.152-b16]
[2019-03-12T07:13:48,051][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] JVM arguments []
[2019-03-12T07:13:48,062][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] no modules loaded
[2019-03-12T07:13:48,063][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [com.automattic.elasticsearch.plugin.StatsdPlugin]
[2019-03-12T07:13:48,064][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
[2019-03-12T07:13:48,064][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin]
[2019-03-12T07:13:48,065][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
[2019-03-12T07:13:48,066][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
[2019-03-12T07:13:48,067][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
[2019-03-12T07:13:48,068][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
[2019-03-12T07:13:48,069][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:13:48,631][WARN ][o.e.d.c.s.Settings       ] [masterFailOverShouldWork] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2019-03-12T07:13:48,634][WARN ][o.e.d.c.s.Settings       ] [masterFailOverShouldWork] [http.pipelining] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2019-03-12T07:13:49,604][INFO ][o.e.d.DiscoveryModule    ] [masterFailOverShouldWork] using discovery type [test-zen] and host providers [settings, file]
[2019-03-12T07:13:50,096][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] initialized
[2019-03-12T07:13:50,105][INFO ][o.e.e.NodeEnvironment    ] [masterFailOverShouldWork] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [129.4gb], net total_space [465.6gb], types [apfs]
[2019-03-12T07:13:50,106][INFO ][o.e.e.NodeEnvironment    ] [masterFailOverShouldWork] heap size [3.5gb], compressed ordinary object pointers [true]
[2019-03-12T07:13:50,107][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] node name [node_s1], node ID [_WPkx6EbRuKB4xqDVW7-dw]
[2019-03-12T07:13:50,108][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] version[6.6.1], pid[87986], build[unknown/unknown/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Mac OS X/10.14.3/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_152/25.152-b16]
[2019-03-12T07:13:50,109][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] JVM arguments []
[2019-03-12T07:13:50,110][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] no modules loaded
[2019-03-12T07:13:50,111][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [com.automattic.elasticsearch.plugin.StatsdPlugin]
[2019-03-12T07:13:50,111][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
[2019-03-12T07:13:50,112][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin]
[2019-03-12T07:13:50,113][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
[2019-03-12T07:13:50,114][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
[2019-03-12T07:13:50,114][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
[2019-03-12T07:13:50,115][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
[2019-03-12T07:13:50,116][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:13:50,137][INFO ][o.e.d.DiscoveryModule    ] [masterFailOverShouldWork] using discovery type [test-zen] and host providers [settings, file]
[2019-03-12T07:13:50,161][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] initialized
[2019-03-12T07:13:50,167][INFO ][o.e.e.NodeEnvironment    ] [masterFailOverShouldWork] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [129.4gb], net total_space [465.6gb], types [apfs]
[2019-03-12T07:13:50,167][INFO ][o.e.e.NodeEnvironment    ] [masterFailOverShouldWork] heap size [3.5gb], compressed ordinary object pointers [true]
[2019-03-12T07:13:50,169][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] node name [node_s2], node ID [mIXn67uLTu2-5LhlK6Qqfg]
[2019-03-12T07:13:50,170][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] version[6.6.1], pid[87986], build[unknown/unknown/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Mac OS X/10.14.3/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_152/25.152-b16]
[2019-03-12T07:13:50,171][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] JVM arguments []
[2019-03-12T07:13:50,172][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] no modules loaded
[2019-03-12T07:13:50,173][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [com.automattic.elasticsearch.plugin.StatsdPlugin]
[2019-03-12T07:13:50,173][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
[2019-03-12T07:13:50,174][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin]
[2019-03-12T07:13:50,175][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
[2019-03-12T07:13:50,176][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
[2019-03-12T07:13:50,177][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
[2019-03-12T07:13:50,177][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
[2019-03-12T07:13:50,178][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:13:50,201][INFO ][o.e.d.DiscoveryModule    ] [masterFailOverShouldWork] using discovery type [test-zen] and host providers [settings, file]
[2019-03-12T07:13:50,224][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] initialized
[2019-03-12T07:13:50,231][INFO ][o.e.n.Node               ] [integ_1] starting ...
[2019-03-12T07:13:50,231][INFO ][o.e.n.Node               ] [integ_3] starting ...
[2019-03-12T07:13:50,231][INFO ][o.e.n.Node               ] [integ_2] starting ...
[2019-03-12T07:13:50,232][INFO ][c.a.e.s.StatsdService    ] [integ_1] StatsD reporting triggered every [1s] to host [localhost:12345] with metric prefix [myhost0]
[2019-03-12T07:13:50,233][INFO ][c.a.e.s.StatsdService    ] [integ_3] StatsD reporting triggered every [1s] to host [localhost:12345] with metric prefix [myhost2]
[2019-03-12T07:13:50,233][INFO ][c.a.e.s.StatsdService    ] [node_s1] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,233][INFO ][c.a.e.s.StatsdService    ] [integ_2] StatsD reporting triggered every [1s] to host [localhost:12345] with metric prefix [myhost1]
[2019-03-12T07:13:50,233][INFO ][c.a.e.s.StatsdService    ] [node_s2] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,233][INFO ][c.a.e.s.StatsdService    ] [node_s0] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,245][INFO ][c.a.e.s.StatsdService    ] [node_s1] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,247][INFO ][c.a.e.s.StatsdService    ] [node_s2] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,248][INFO ][c.a.e.s.StatsdService    ] [node_s0] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,256][INFO ][c.a.e.s.StatsdService    ] [node_s1] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,258][INFO ][c.a.e.s.StatsdService    ] [node_s2] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,259][INFO ][c.a.e.s.StatsdService    ] [node_s0] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,265][INFO ][o.e.t.TransportService   ] [integ_2] publish_address {127.0.0.1:64745}, bound_addresses {[::1]:64740}, {127.0.0.1:64745}
[2019-03-12T07:13:50,265][INFO ][o.e.t.TransportService   ] [integ_3] publish_address {127.0.0.1:64743}, bound_addresses {[::1]:64741}, {127.0.0.1:64743}
[2019-03-12T07:13:50,265][INFO ][o.e.t.TransportService   ] [integ_1] publish_address {127.0.0.1:64744}, bound_addresses {[::1]:64742}, {127.0.0.1:64744}
[2019-03-12T07:13:50,267][INFO ][c.a.e.s.StatsdService    ] [node_s1] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,269][INFO ][c.a.e.s.StatsdService    ] [node_s2] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,271][INFO ][c.a.e.s.StatsdService    ] [node_s0] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,281][INFO ][c.a.e.s.StatsdService    ] [node_s1] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,281][INFO ][c.a.e.s.StatsdService    ] [node_s2] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,284][INFO ][c.a.e.s.StatsdService    ] [node_s0] Cluster state not set yet. Looping...
[2019-03-12T07:13:50,300][INFO ][o.e.t.d.MockZenPing      ] [node_s0] pinging using mock zen ping
[2019-03-12T07:13:50,300][INFO ][o.e.t.d.MockZenPing      ] [node_s2] pinging using mock zen ping
[2019-03-12T07:13:50,300][INFO ][o.e.t.d.MockZenPing      ] [node_s1] pinging using mock zen ping
[2019-03-12T07:13:50,408][INFO ][o.e.c.s.MasterService    ] [node_s1] zen-disco-elected-as-master ([1] nodes joined)[{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}], zen-disco-node-join[{node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}], reason: new_master {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}, added {{node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743},{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744},}
[2019-03-12T07:13:50,430][INFO ][o.e.c.s.ClusterApplierService] [node_s0] detected_master {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}, added {{node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743},{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745},}, reason: apply cluster state (from master [master {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745} committed version [1]])
[2019-03-12T07:13:50,430][INFO ][o.e.c.s.ClusterApplierService] [node_s2] detected_master {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}, added {{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745},{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744},}, reason: apply cluster state (from master [master {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745} committed version [1]])
[2019-03-12T07:13:50,443][INFO ][o.e.n.Node               ] [integ_3] started
[2019-03-12T07:13:50,444][INFO ][o.e.n.Node               ] [integ_1] started
[2019-03-12T07:13:50,445][INFO ][o.e.c.s.ClusterApplierService] [node_s1] new_master {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}, added {{node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743},{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744},}, reason: apply cluster state (from master [master {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745} committed version [1] source [zen-disco-elected-as-master ([1] nodes joined)[{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}], zen-disco-node-join[{node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}]]])
[2019-03-12T07:13:50,467][INFO ][o.e.n.Node               ] [integ_2] started
[2019-03-12T07:13:50,513][INFO ][o.e.g.GatewayService     ] [node_s1] recovered [0] indices into cluster_state
[2019-03-12T07:13:50,521][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] no modules loaded
[2019-03-12T07:13:50,522][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:13:50,562][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] no modules loaded
[2019-03-12T07:13:50,563][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:13:50,784][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s1] adding template [random_index_template] for index patterns [*]
[2019-03-12T07:13:50,794][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [masterFailOverShouldWork] [StatsdPluginIntegrationTest#masterFailOverShouldWork]: all set up test
[2019-03-12T07:13:50,795][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [masterFailOverShouldWork] Creating index e5c2d7
[2019-03-12T07:13:50,796][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] no modules loaded
[2019-03-12T07:13:50,797][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:13:50,830][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s1] [e5c2d7] creating index, cause [api], templates [random_index_template], shards [4]/[1], mappings []
[2019-03-12T07:13:51,491][INFO ][o.e.c.r.a.AllocationService] [node_s1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[e5c2d7][2], [e5c2d7][3], [e5c2d7][0]] ...]).
[2019-03-12T07:13:51,555][INFO ][o.e.c.m.MetaDataMappingService] [node_s1] [e5c2d7/XQ-yd898TtG5830bRHL10g] create_mapping [5e077d]
[2019-03-12T07:13:51,991][INFO ][o.e.t.InternalTestCluster] [masterFailOverShouldWork] Closing master node [node_s1] 
[2019-03-12T07:13:52,038][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] stopping ...
[2019-03-12T07:13:52,043][INFO ][o.e.d.z.ZenDiscovery     ] [node_s0] master_left [{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}], reason [shut_down]
[2019-03-12T07:13:52,043][INFO ][o.e.d.z.ZenDiscovery     ] [node_s2] master_left [{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}], reason [shut_down]
[2019-03-12T07:13:52,045][ERROR][c.a.e.s.StatsdService    ] [node_s1] Exiting StatsdReporterThread
[2019-03-12T07:13:52,045][INFO ][c.a.e.s.StatsdService    ] [masterFailOverShouldWork] StatsD reporter stopped
[2019-03-12T07:13:52,044][INFO ][o.e.d.z.ZenDiscovery     ] [node_s2] master_left [{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}], reason [transport disconnected]
[2019-03-12T07:13:52,044][INFO ][o.e.d.z.ZenDiscovery     ] [node_s0] master_left [{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}], reason [transport disconnected]
[2019-03-12T07:13:52,048][WARN ][o.e.d.z.ZenDiscovery     ] [node_s2] master left (reason = transport disconnected), current nodes: nodes: 
   {node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}, local
   {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}, master
   {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}

[2019-03-12T07:13:52,046][WARN ][o.e.d.z.ZenDiscovery     ] [node_s0] master left (reason = shut_down), current nodes: nodes: 
   {node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}
   {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}, master
   {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}, local

[2019-03-12T07:13:52,055][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [e5c2d7][0] start check index
[2019-03-12T07:13:52,051][INFO ][o.e.t.d.MockZenPing      ] [node_s2] pinging using mock zen ping
[2019-03-12T07:13:52,055][INFO ][o.e.t.d.MockZenPing      ] [node_s0] pinging using mock zen ping
[2019-03-12T07:13:52,056][WARN ][o.e.t.TcpTransport       ] [node_s0] send message failed [channel: MockChannel{profile='default', isOpen=false, localAddress=/127.0.0.1:64749, isServerSocket=false}]
java.net.SocketException: Socket is closed
	at java.net.Socket.getOutputStream(Socket.java:943) ~[?:1.8.0_152]
	at org.elasticsearch.transport.MockTcpTransport$MockChannel.sendMessage(MockTcpTransport.java:437) [framework-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:760) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:847) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:817) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:64) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:291) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:286) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1288) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
[2019-03-12T07:13:52,053][WARN ][o.e.t.TcpTransport       ] [node_s2] send message failed [channel: MockChannel{profile='default', isOpen=false, localAddress=/127.0.0.1:64748, isServerSocket=false}]
java.net.SocketException: Socket is closed
	at java.net.Socket.getOutputStream(Socket.java:943) ~[?:1.8.0_152]
	at org.elasticsearch.transport.MockTcpTransport$MockChannel.sendMessage(MockTcpTransport.java:437) [framework-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:760) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:847) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:817) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:64) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:291) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:286) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1288) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
[2019-03-12T07:13:52,055][WARN ][o.e.c.NodeConnectionsService] [node_s2] failed to connect to node {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [node_s1][127.0.0.1:64745] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1569) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-6.6.1.jar:6.6.1]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_152]
	at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.MockTcpTransport.lambda$initiateChannel$0(MockTcpTransport.java:195) ~[framework-6.6.1.jar:6.6.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_152]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_152]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_152]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_152]
	at org.elasticsearch.mocksocket.MockSocket.access$101(MockSocket.java:32) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.mocksocket.MockSocket.lambda$connect$0(MockSocket.java:66) ~[mocksocket-1.2.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at org.elasticsearch.mocksocket.MockSocket.connect(MockSocket.java:65) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.mocksocket.MockSocket.connect(MockSocket.java:59) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.transport.MockTcpTransport.lambda$initiateChannel$0(MockTcpTransport.java:190) ~[framework-6.6.1.jar:6.6.1]
	... 5 more
[2019-03-12T07:13:52,057][WARN ][o.e.c.NodeConnectionsService] [node_s0] failed to connect to node {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [node_s1][127.0.0.1:64745] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1569) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-6.6.1.jar:6.6.1]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_152]
	at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.MockTcpTransport.lambda$initiateChannel$0(MockTcpTransport.java:195) ~[framework-6.6.1.jar:6.6.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_152]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_152]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_152]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_152]
	at org.elasticsearch.mocksocket.MockSocket.access$101(MockSocket.java:32) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.mocksocket.MockSocket.lambda$connect$0(MockSocket.java:66) ~[mocksocket-1.2.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at org.elasticsearch.mocksocket.MockSocket.connect(MockSocket.java:65) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.mocksocket.MockSocket.connect(MockSocket.java:59) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.transport.MockTcpTransport.lambda$initiateChannel$0(MockTcpTransport.java:190) ~[framework-6.6.1.jar:6.6.1]
	... 5 more
[2019-03-12T07:13:52,097][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [e5c2d7][0] end check index
[2019-03-12T07:13:52,106][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [e5c2d7][2] start check index
[2019-03-12T07:13:52,108][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [e5c2d7][2] end check index
[2019-03-12T07:13:52,111][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [e5c2d7][3] start check index
[2019-03-12T07:13:52,113][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [e5c2d7][3] end check index
[2019-03-12T07:13:52,114][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] stopped
[2019-03-12T07:13:52,114][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] closing ...
[2019-03-12T07:13:52,121][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] closed
[2019-03-12T07:13:52,132][INFO ][o.e.e.NodeEnvironment    ] [masterFailOverShouldWork] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [129.4gb], net total_space [465.6gb], types [apfs]
[2019-03-12T07:13:52,133][INFO ][o.e.e.NodeEnvironment    ] [masterFailOverShouldWork] heap size [3.5gb], compressed ordinary object pointers [true]
[2019-03-12T07:13:52,142][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] node name [node_s3], node ID [_WPkx6EbRuKB4xqDVW7-dw]
[2019-03-12T07:13:52,143][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] version[6.6.1], pid[87986], build[unknown/unknown/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Mac OS X/10.14.3/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_152/25.152-b16]
[2019-03-12T07:13:52,144][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] JVM arguments []
[2019-03-12T07:13:52,145][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] no modules loaded
[2019-03-12T07:13:52,145][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [com.automattic.elasticsearch.plugin.StatsdPlugin]
[2019-03-12T07:13:52,146][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
[2019-03-12T07:13:52,146][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin]
[2019-03-12T07:13:52,147][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
[2019-03-12T07:13:52,148][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
[2019-03-12T07:13:52,148][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
[2019-03-12T07:13:52,149][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
[2019-03-12T07:13:52,149][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:13:52,171][INFO ][o.e.d.DiscoveryModule    ] [masterFailOverShouldWork] using discovery type [test-zen] and host providers [settings, file]
[2019-03-12T07:13:52,189][INFO ][o.e.n.Node               ] [masterFailOverShouldWork] initialized
[2019-03-12T07:13:55,066][INFO ][o.e.c.s.MasterService    ] [node_s0] zen-disco-elected-as-master ([1] nodes joined)[{node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}], reason: new_master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}
[2019-03-12T07:13:55,070][INFO ][o.e.c.s.ClusterApplierService] [node_s2] detected_master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [11]])
[2019-03-12T07:13:55,074][WARN ][o.e.d.z.PublishClusterStateAction] [node_s0] publishing cluster state with version [11] failed for the following nodes: [[{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}]]
[2019-03-12T07:13:55,075][INFO ][o.e.c.s.ClusterApplierService] [node_s0] new_master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [11] source [zen-disco-elected-as-master ([1] nodes joined)[{node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}]]])
[2019-03-12T07:13:55,091][INFO ][o.e.c.s.MasterService    ] [node_s0] zen-disco-node-failed({node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}), reason(transport disconnected)[{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745} transport disconnected], reason: removed {{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745},}
[2019-03-12T07:13:55,096][INFO ][o.e.c.s.ClusterApplierService] [node_s2] removed {{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [12]])
[2019-03-12T07:13:55,106][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [12] source [zen-disco-node-failed({node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745}), reason(transport disconnected)[{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{cgrSdlF9Tn-nQIk2NP5kgw}{127.0.0.1}{127.0.0.1:64745} transport disconnected]]])
[2019-03-12T07:13:55,118][INFO ][o.e.i.s.IndexShard       ] [node_s2] [e5c2d7][3] primary-replica resync completed with 0 operations
[2019-03-12T07:13:55,120][INFO ][o.e.i.s.IndexShard       ] [node_s0] [e5c2d7][0] primary-replica resync completed with 0 operations
[2019-03-12T07:13:55,121][INFO ][o.e.n.Node               ] [integ_4] starting ...
[2019-03-12T07:13:55,122][INFO ][c.a.e.s.StatsdService    ] [integ_4] StatsD reporting triggered every [1s] to host [localhost:12345] with metric prefix [myhost3]
[2019-03-12T07:13:55,122][INFO ][c.a.e.s.StatsdService    ] [node_s3] Cluster state not set yet. Looping...
[2019-03-12T07:13:55,128][INFO ][o.e.t.TransportService   ] [integ_4] publish_address {127.0.0.1:64773}, bound_addresses {[::1]:64772}, {127.0.0.1:64773}
[2019-03-12T07:13:55,135][INFO ][o.e.t.d.MockZenPing      ] [node_s3] pinging using mock zen ping
[2019-03-12T07:13:55,156][INFO ][o.e.c.s.MasterService    ] [node_s0] zen-disco-node-join[{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773}], reason: added {{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773},}
[2019-03-12T07:13:55,161][INFO ][o.e.c.s.ClusterApplierService] [node_s2] added {{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [14]])
[2019-03-12T07:13:55,161][INFO ][o.e.c.s.ClusterApplierService] [node_s3] detected_master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}, added {{node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743},{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [14]])
[2019-03-12T07:13:55,180][INFO ][o.e.n.Node               ] [integ_4] started
[2019-03-12T07:13:55,181][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [14] source [zen-disco-node-join[{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773}]]])
[2019-03-12T07:13:55,182][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] no modules loaded
[2019-03-12T07:13:55,185][INFO ][o.e.p.PluginsService     ] [masterFailOverShouldWork] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:13:55,282][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[e5c2d7][0]] ...]).
[2019-03-12T07:13:55,380][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [e5c2d7][1] start check index
[2019-03-12T07:13:55,384][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [e5c2d7][1] end check index
[2019-03-12T07:13:55,395][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [e5c2d7][0] start check index
[2019-03-12T07:13:55,401][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [e5c2d7][0] end check index
stopped master
[2019-03-12T07:14:03,237][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [masterFailOverShouldWork] [StatsdPluginIntegrationTest#masterFailOverShouldWork]: cleaning up after test
[2019-03-12T07:14:03,303][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [e5c2d7/XQ-yd898TtG5830bRHL10g] deleting index
[2019-03-12T07:14:03,310][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [e5c2d7][1] start check index
[2019-03-12T07:14:03,311][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s3] [e5c2d7][0] start check index
[2019-03-12T07:14:03,314][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [e5c2d7][1] end check index
[2019-03-12T07:14:03,318][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s3] [e5c2d7][0] end check index
[2019-03-12T07:14:03,328][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s3] [e5c2d7][1] start check index
[2019-03-12T07:14:03,332][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s3] [e5c2d7][1] end check index
[2019-03-12T07:14:03,332][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [e5c2d7][2] start check index
[2019-03-12T07:14:03,339][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [e5c2d7][2] end check index
[2019-03-12T07:14:03,351][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [e5c2d7][3] start check index
[2019-03-12T07:14:03,354][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [e5c2d7][3] end check index
[2019-03-12T07:14:03,368][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [e5c2d7][0] start check index
[2019-03-12T07:14:03,376][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [e5c2d7][0] end check index
[2019-03-12T07:14:03,388][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [e5c2d7][2] start check index
[2019-03-12T07:14:03,392][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [e5c2d7][2] end check index
[2019-03-12T07:14:03,399][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [e5c2d7][3] start check index
[2019-03-12T07:14:03,401][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [e5c2d7][3] end check index
[2019-03-12T07:14:03,414][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template]
[2019-03-12T07:14:03,426][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [masterFailOverShouldWork] [StatsdPluginIntegrationTest#masterFailOverShouldWork]: cleaned up after test
[2019-03-12T07:14:03,428][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [masterFailOverShouldWork] after test
[2019-03-12T07:14:03,450][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [testThatIndexingResultsInMonitoring] before test
[2019-03-12T07:14:03,451][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [testThatIndexingResultsInMonitoring] [StatsdPluginIntegrationTest#testThatIndexingResultsInMonitoring]: setting up test
[2019-03-12T07:14:03,455][INFO ][o.e.n.Node               ] [testThatIndexingResultsInMonitoring] stopping ...
[2019-03-12T07:14:03,457][INFO ][o.e.c.s.MasterService    ] [node_s0] zen-disco-node-left({node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773}), reason(left)[{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773} left], reason: removed {{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773},}
[2019-03-12T07:14:03,458][ERROR][c.a.e.s.StatsdService    ] [node_s3] Exiting StatsdReporterThread
[2019-03-12T07:14:03,458][INFO ][c.a.e.s.StatsdService    ] [testThatIndexingResultsInMonitoring] StatsD reporter stopped
[2019-03-12T07:14:03,460][INFO ][o.e.c.s.ClusterApplierService] [node_s2] removed {{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [22]])
[2019-03-12T07:14:03,460][INFO ][o.e.n.Node               ] [testThatIndexingResultsInMonitoring] stopped
[2019-03-12T07:14:03,462][INFO ][o.e.n.Node               ] [testThatIndexingResultsInMonitoring] closing ...
[2019-03-12T07:14:03,462][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [22] source [zen-disco-node-left({node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773}), reason(left)[{node_s3}{_WPkx6EbRuKB4xqDVW7-dw}{exCDwqslT8y0Bhl1KAhyDg}{127.0.0.1}{127.0.0.1:64773} left]]])
[2019-03-12T07:14:03,464][INFO ][o.e.n.Node               ] [testThatIndexingResultsInMonitoring] closed
[2019-03-12T07:14:03,466][INFO ][o.e.t.InternalTestCluster] [testThatIndexingResultsInMonitoring] Successfully wiped data directory for node location: /private/var/folders/sl/pl51hwc91zl8x2x53cs5sbm80000gn/T/com.automattic.elasticsearch.statsd.test.StatsdPluginIntegrationTest_D8C8958638594349-001/tempDir-002/data/nodes/1
[2019-03-12T07:14:03,471][INFO ][o.e.e.NodeEnvironment    ] [testThatIndexingResultsInMonitoring] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [129.4gb], net total_space [465.6gb], types [apfs]
[2019-03-12T07:14:03,472][INFO ][o.e.e.NodeEnvironment    ] [testThatIndexingResultsInMonitoring] heap size [3.5gb], compressed ordinary object pointers [true]
[2019-03-12T07:14:03,473][INFO ][o.e.n.Node               ] [testThatIndexingResultsInMonitoring] node name [node_s1], node ID [_WPkx6EbRuKB4xqDVW7-dw]
[2019-03-12T07:14:03,473][INFO ][o.e.n.Node               ] [testThatIndexingResultsInMonitoring] version[6.6.1], pid[87986], build[unknown/unknown/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Mac OS X/10.14.3/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_152/25.152-b16]
[2019-03-12T07:14:03,474][INFO ][o.e.n.Node               ] [testThatIndexingResultsInMonitoring] JVM arguments []
[2019-03-12T07:14:03,475][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] no modules loaded
[2019-03-12T07:14:03,476][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] loaded plugin [com.automattic.elasticsearch.plugin.StatsdPlugin]
[2019-03-12T07:14:03,477][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
[2019-03-12T07:14:03,478][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] loaded plugin [org.elasticsearch.search.MockSearchService$TestPlugin]
[2019-03-12T07:14:03,478][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
[2019-03-12T07:14:03,479][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
[2019-03-12T07:14:03,480][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
[2019-03-12T07:14:03,481][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
[2019-03-12T07:14:03,482][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:14:03,499][INFO ][o.e.d.DiscoveryModule    ] [testThatIndexingResultsInMonitoring] using discovery type [test-zen] and host providers [settings, file]
[2019-03-12T07:14:03,514][INFO ][o.e.n.Node               ] [testThatIndexingResultsInMonitoring] initialized
[2019-03-12T07:14:03,518][INFO ][o.e.n.Node               ] [integ_6] starting ...
[2019-03-12T07:14:03,519][INFO ][c.a.e.s.StatsdService    ] [integ_6] StatsD reporting triggered every [1s] to host [localhost:12345] with metric prefix [myhost1]
[2019-03-12T07:14:03,519][INFO ][c.a.e.s.StatsdService    ] [node_s1] Cluster state not set yet. Looping...
[2019-03-12T07:14:03,523][INFO ][o.e.t.TransportService   ] [integ_6] publish_address {127.0.0.1:64808}, bound_addresses {[::1]:64807}, {127.0.0.1:64808}
[2019-03-12T07:14:03,526][INFO ][o.e.t.d.MockZenPing      ] [node_s1] pinging using mock zen ping
[2019-03-12T07:14:03,531][INFO ][o.e.c.s.MasterService    ] [node_s0] zen-disco-node-join[{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{Elu90p28QdKi0OPcArEtzQ}{127.0.0.1}{127.0.0.1:64808}], reason: added {{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{Elu90p28QdKi0OPcArEtzQ}{127.0.0.1}{127.0.0.1:64808},}
[2019-03-12T07:14:03,535][INFO ][o.e.c.s.ClusterApplierService] [node_s2] added {{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{Elu90p28QdKi0OPcArEtzQ}{127.0.0.1}{127.0.0.1:64808},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [23]])
[2019-03-12T07:14:03,535][INFO ][o.e.c.s.ClusterApplierService] [node_s1] detected_master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}, added {{node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743},{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [23]])
[2019-03-12T07:14:03,543][INFO ][o.e.n.Node               ] [integ_6] started
[2019-03-12T07:14:03,543][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{Elu90p28QdKi0OPcArEtzQ}{127.0.0.1}{127.0.0.1:64808},}, reason: apply cluster state (from master [master {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} committed version [23] source [zen-disco-node-join[{node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{Elu90p28QdKi0OPcArEtzQ}{127.0.0.1}{127.0.0.1:64808}]]])
[2019-03-12T07:14:03,552][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*]
[2019-03-12T07:14:03,558][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [testThatIndexingResultsInMonitoring] [StatsdPluginIntegrationTest#testThatIndexingResultsInMonitoring]: all set up test
[2019-03-12T07:14:03,559][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [testThatIndexingResultsInMonitoring] Creating index 99aa0e
[2019-03-12T07:14:03,562][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [99aa0e] creating index, cause [api], templates [random_index_template], shards [4]/[1], mappings []
[2019-03-12T07:14:03,699][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [99aa0e/F6-vHkzoQEy3_UR4Ti4Zbg] create_mapping [5e077d]
[2019-03-12T07:14:03,751][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] no modules loaded
[2019-03-12T07:14:03,752][INFO ][o.e.p.PluginsService     ] [testThatIndexingResultsInMonitoring] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
[2019-03-12T07:14:03,828][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[99aa0e][1]] ...]).
[2019-03-12T07:14:06,235][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [testThatIndexingResultsInMonitoring] [StatsdPluginIntegrationTest#testThatIndexingResultsInMonitoring]: cleaning up after test
[2019-03-12T07:14:06,258][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [99aa0e/F6-vHkzoQEy3_UR4Ti4Zbg] deleting index
[2019-03-12T07:14:06,263][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [99aa0e][0] start check index
[2019-03-12T07:14:06,263][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [99aa0e][0] start check index
[2019-03-12T07:14:06,283][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [99aa0e][0] end check index
[2019-03-12T07:14:06,283][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [99aa0e][0] end check index
[2019-03-12T07:14:06,291][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [99aa0e][3] start check index
[2019-03-12T07:14:06,291][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [99aa0e][1] start check index
[2019-03-12T07:14:06,300][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s1] [99aa0e][3] end check index
[2019-03-12T07:14:06,307][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [99aa0e][1] end check index
[2019-03-12T07:14:06,315][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [99aa0e][2] start check index
[2019-03-12T07:14:06,321][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s2] [99aa0e][2] end check index
[2019-03-12T07:14:06,330][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [99aa0e][1] start check index
[2019-03-12T07:14:06,336][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [99aa0e][1] end check index
[2019-03-12T07:14:06,344][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [99aa0e][2] start check index
[2019-03-12T07:14:06,349][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [99aa0e][2] end check index
[2019-03-12T07:14:06,355][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [99aa0e][3] start check index
[2019-03-12T07:14:06,361][INFO ][o.e.t.s.MockFSIndexStore$Listener] [node_s0] [99aa0e][3] end check index
[2019-03-12T07:14:06,374][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template]
[2019-03-12T07:14:06,382][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [testThatIndexingResultsInMonitoring] [StatsdPluginIntegrationTest#testThatIndexingResultsInMonitoring]: cleaned up after test
[2019-03-12T07:14:06,383][INFO ][c.a.e.s.t.StatsdPluginIntegrationTest] [testThatIndexingResultsInMonitoring] after test
Waiting for cleanup
java.net.SocketException: Socket closed
	at java.net.PlainDatagramSocketImpl.peekData(Native Method)
	at java.net.DatagramSocket.receive(DatagramSocket.java:743)
[2019-03-12T07:14:16,392][INFO ][o.e.n.Node               ] [suite] stopping ...
	at com.automattic.elasticsearch.statsd.test.StatsdMockServer.run(StatsdMockServer.java:40)
[2019-03-12T07:14:16,395][INFO ][o.e.d.z.ZenDiscovery     ] [node_s2] master_left [{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}], reason [shut_down]
[2019-03-12T07:14:16,397][INFO ][o.e.d.z.ZenDiscovery     ] [node_s2] master_left [{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}], reason [transport disconnected]
[2019-03-12T07:14:16,395][INFO ][o.e.d.z.ZenDiscovery     ] [node_s1] master_left [{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}], reason [shut_down]
[2019-03-12T07:14:16,397][INFO ][o.e.d.z.ZenDiscovery     ] [node_s1] master_left [{node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}], reason [transport disconnected]
[2019-03-12T07:14:16,397][INFO ][c.a.e.s.StatsdService    ] [suite] StatsD reporter stopped
[2019-03-12T07:14:16,397][WARN ][o.e.d.z.ZenDiscovery     ] [node_s2] master left (reason = shut_down), current nodes: nodes: 
   {node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}, local
   {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{Elu90p28QdKi0OPcArEtzQ}{127.0.0.1}{127.0.0.1:64808}
   {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}, master

[2019-03-12T07:14:16,397][ERROR][c.a.e.s.StatsdService    ] [node_s0] Exiting StatsdReporterThread
[2019-03-12T07:14:16,400][WARN ][o.e.d.z.ZenDiscovery     ] [node_s1] master left (reason = shut_down), current nodes: nodes: 
   {node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}
   {node_s1}{_WPkx6EbRuKB4xqDVW7-dw}{Elu90p28QdKi0OPcArEtzQ}{127.0.0.1}{127.0.0.1:64808}, local
   {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744}, master

[2019-03-12T07:14:16,402][INFO ][o.e.n.Node               ] [suite] stopped
[2019-03-12T07:14:16,408][INFO ][o.e.n.Node               ] [suite] closing ...
[2019-03-12T07:14:16,407][INFO ][o.e.t.d.MockZenPing      ] [node_s1] pinging using mock zen ping
[2019-03-12T07:14:16,406][WARN ][o.e.c.NodeConnectionsService] [node_s2] failed to connect to node {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [node_s0][127.0.0.1:64744] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1569) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-6.6.1.jar:6.6.1]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_152]
	at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.MockTcpTransport.lambda$initiateChannel$0(MockTcpTransport.java:195) ~[framework-6.6.1.jar:6.6.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_152]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_152]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_152]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_152]
	at org.elasticsearch.mocksocket.MockSocket.access$101(MockSocket.java:32) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.mocksocket.MockSocket.lambda$connect$0(MockSocket.java:66) ~[mocksocket-1.2.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at org.elasticsearch.mocksocket.MockSocket.connect(MockSocket.java:65) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.mocksocket.MockSocket.connect(MockSocket.java:59) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.transport.MockTcpTransport.lambda$initiateChannel$0(MockTcpTransport.java:190) ~[framework-6.6.1.jar:6.6.1]
	... 5 more
[2019-03-12T07:14:16,404][WARN ][o.e.t.TcpTransport       ] [node_s2] send message failed [channel: MockChannel{profile='default', isOpen=false, localAddress=/127.0.0.1:64750, isServerSocket=false}]
java.net.SocketException: Socket is closed
	at java.net.Socket.getOutputStream(Socket.java:943) ~[?:1.8.0_152]
	at org.elasticsearch.transport.MockTcpTransport$MockChannel.sendMessage(MockTcpTransport.java:437) [framework-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:760) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:847) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:817) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:64) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:291) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:286) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1288) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
[2019-03-12T07:14:16,404][INFO ][o.e.t.d.MockZenPing      ] [node_s2] pinging using mock zen ping
[2019-03-12T07:14:16,411][INFO ][o.e.n.Node               ] [suite] closed
[2019-03-12T07:14:16,408][WARN ][o.e.c.NodeConnectionsService] [node_s1] failed to connect to node {node_s0}{lmEpwuT6R0GleHXKBBYH6A}{DP9mjxVAQ02lllMCPFMbZw}{127.0.0.1}{127.0.0.1:64744} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [node_s0][127.0.0.1:64744] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1569) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-6.6.1.jar:6.6.1]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_152]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_152]
	at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.MockTcpTransport.lambda$initiateChannel$0(MockTcpTransport.java:195) ~[framework-6.6.1.jar:6.6.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_152]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_152]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_152]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_152]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_152]
	at org.elasticsearch.mocksocket.MockSocket.access$101(MockSocket.java:32) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.mocksocket.MockSocket.lambda$connect$0(MockSocket.java:66) ~[mocksocket-1.2.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at org.elasticsearch.mocksocket.MockSocket.connect(MockSocket.java:65) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.mocksocket.MockSocket.connect(MockSocket.java:59) ~[mocksocket-1.2.jar:?]
	at org.elasticsearch.transport.MockTcpTransport.lambda$initiateChannel$0(MockTcpTransport.java:190) ~[framework-6.6.1.jar:6.6.1]
	... 5 more
[2019-03-12T07:14:16,407][WARN ][o.e.t.TcpTransport       ] [node_s1] send message failed [channel: MockChannel{profile='default', isOpen=false, localAddress=/127.0.0.1:64810, isServerSocket=false}]
java.net.SocketException: Socket is closed
	at java.net.Socket.getOutputStream(Socket.java:943) ~[?:1.8.0_152]
	at org.elasticsearch.transport.MockTcpTransport$MockChannel.sendMessage(MockTcpTransport.java:437) [framework-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:760) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:847) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:817) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:64) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:291) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.discovery.zen.MembershipAction$LeaveRequestRequestHandler.messageReceived(MembershipAction.java:286) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1288) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.1.jar:6.6.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
[2019-03-12T07:14:16,437][INFO ][o.e.n.Node               ] [suite] stopping ...
[2019-03-12T07:14:16,459][WARN ][o.e.d.z.ZenDiscovery     ] [node_s2] not enough master nodes discovered during pinging (found [[Candidate{node={node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}, clusterStateVersion=33}]], but needed [2]), pinging again
[2019-03-12T07:14:16,459][ERROR][c.a.e.s.StatsdService    ] [node_s1] Exiting StatsdReporterThread
[2019-03-12T07:14:16,459][INFO ][c.a.e.s.StatsdService    ] [suite] StatsD reporter stopped
[2019-03-12T07:14:16,460][INFO ][o.e.t.d.MockZenPing      ] [node_s2] pinging using mock zen ping
[2019-03-12T07:14:16,461][INFO ][o.e.n.Node               ] [suite] stopped
[2019-03-12T07:14:16,461][INFO ][o.e.n.Node               ] [suite] closing ...
[2019-03-12T07:14:16,463][INFO ][o.e.n.Node               ] [suite] closed
[2019-03-12T07:14:16,464][INFO ][o.e.n.Node               ] [suite] stopping ...
[2019-03-12T07:14:16,465][WARN ][o.e.d.z.ZenDiscovery     ] [node_s2] not enough master nodes discovered during pinging (found [[Candidate{node={node_s2}{mIXn67uLTu2-5LhlK6Qqfg}{YkpWFSKwQ7m6rItIgIpTUQ}{127.0.0.1}{127.0.0.1:64743}, clusterStateVersion=33}]], but needed [2]), pinging again
[2019-03-12T07:14:16,466][ERROR][c.a.e.s.StatsdService    ] [node_s2] Exiting StatsdReporterThread
[2019-03-12T07:14:16,466][INFO ][c.a.e.s.StatsdService    ] [suite] StatsD reporter stopped
[2019-03-12T07:14:16,467][INFO ][o.e.n.Node               ] [suite] stopped
[2019-03-12T07:14:16,468][INFO ][o.e.n.Node               ] [suite] closing ...
[2019-03-12T07:14:16,469][INFO ][o.e.n.Node               ] [suite] closed
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.944 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ elasticsearch-statsd ---
[INFO] Building jar: /Users/murilo/mesosphere/elasticsearch-statsd-plugin/target/elasticsearch-statsd-6.6.1.0.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.3:single (default) @ elasticsearch-statsd ---
[INFO] Reading assembly descriptor: /Users/murilo/mesosphere/elasticsearch-statsd-plugin/src/main/assemblies/plugin.xml
[WARNING] The following patterns were never triggered in this artifact exclusion filter:
o  'org.elasticsearch:elasticsearch'

[INFO] Building zip: /Users/murilo/mesosphere/elasticsearch-statsd-plugin/target/releases/elasticsearch-statsd-6.6.1.0.zip
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  35.635 s
[INFO] Finished at: 2019-03-11T20:14:18+01:00
[INFO] ------------------------------------------------------------------------

mpereira added 3 commits March 11, 2019 20:16
@mpereira
Bump Elasticsearch library version to 6.6.1.
70e056f
@mpereira
Don't use settings field from AbstractLifecycleComponent.
2a8fd78
@mpereira
Don't depend on AbstractLifecycleComponent's logger.
c9baf4d 
`AbstractLifecycleComponent` doesn't extend `AbstractComponent`
anymore, so the logging facilities are unavailable. The changes in the
Elasticsearch codebase now make classes instantiate their own logger
objects.

See more details in
elastic/elasticsearch#35560.
@mpereira mpereira requested a review from phrynchanka June 20, 2019 16:25
phrynchanka
phrynchanka approved these changes Jun 21, 2019
View reviewed changes
Copy link

@phrynchanka phrynchanka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@mpereira mpereira merged commit 1070197 into master Jun 21, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@phrynchanka phrynchanka phrynchanka approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mpereira @phrynchanka