Seatunnel部署

🍑一、概要


安装包及源码包下载地址
在这里插入图片描述

在这里插入图片描述

Apache SeaTunnel 是一个分布式、高性能、易扩展的数据集成平台,用于实时和离线数据处理。它支持多种数据源之间的数据迁移和转换。其中, apache-seatunnel-web-1.0.0-bin.tar.gzapache-seatunnel-2.3.3-bin.tar.gz 分别代表了 Apache SeaTunnel Web 界面的安装包和 Apache SeaTunnel 的核心引擎的安装包。

  • apache-seatunnel-web-1.0.0-bin.tar.gz :这是 Apache SeaTunnel Web 的安装包,提供了用户友好的 Web 界面,使得用户可以更方便地 管理和监控 SeaTunnel 任务 ,包括创建、编辑和运行作业等操作。通过这个界面,用户可以无需直接接触命令行就能进行配置和管理。

  • apache-seatunnel-2.3.3-bin.tar.gz :这是 Apache SeaTunnel 核心引擎的二进制发行版,包含了执行 数据抽取、转换和加载的核心代码库 。它是实际执行数据处理任务的部分,可以与各种数据源和目标(如数据库、文件系统、消息队列等)进行交互。

在部署 Apache SeaTunnel 时,通常需要先安装并配置好核心引擎,然后根据需要选择是否使用 Web 界面进行管理。如果你 只需要在命令行下运行和管理任务 ,那么 只需要安装核心引擎 即可。如果你 想拥有一个图形化的任务管理界面 ,那么就 需要同时安装 Apache SeaTunnel Web

🍑二、核心引擎部署


🍊上传解压安装包: tar xf apache-seatunnel-2.3.3-bin.tar.gz
在这里插入图片描述

🍊 自动安装

🍓自动安装插件(全量安装,很慢,一直到下班,一个半小时都没结束,第二天早上来显示下载结束)

1
2
3
4
5
6
7
[opensource@bigdata02 bin]$ pwd
/home/opensource/app/apache-seatunnel-2.3.3/bin
[opensource@bigdata02 bin]$ ls
install-plugin.sh seatunnel.sh start-seatunnel-flink-15-connector-v2.sh start-seatunnel-spark-3-connector-v2.sh
seatunnel-cluster.sh start-seatunnel-flink-13-connector-v2.sh start-seatunnel-spark-2-connector-v2.sh stop-seatunnel-cluster.sh
[opensource@bigdata02 bin]$
[opensource@bigdata02 bin]$ ./install-plugin.sh

在这里插入图片描述

🍓 验证服务

🍒执行系统自带的任务进行验证: sh seatunnel.sh --config ../config/v2.batch.config.template -m local
在这里插入图片描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
[opensource@bigdata02 bin]$ sh seatunnel.sh --config ../config/v2.batch.config.template -m local 
十二月 08, 2023 9:09:51 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
十二月 08, 2023 9:09:51 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml
十二月 08, 2023 9:09:51 上午 org.apache.seatunnel.engine.common.config.SeaTunnelConfig
信息: seatunnel.home is /home/opensource/app/apache-seatunnel-2.3.3/bin
十二月 08, 2023 9:09:51 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
十二月 08, 2023 9:09:51 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml
2023-12-08 09:09:51,483 WARN com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-233929] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2023-12-08 09:09:51,487 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-233929] [5.1] Resolving domain name 'localhost' to address(es): [127.0.0.1]
2023-12-08 09:09:51,487 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-233929] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [localhost/127.0.0.1]
2023-12-08 09:09:51,509 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - SeaTunnel server start...
2023-12-08 09:09:51,510 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2023-12-08 09:09:51,510 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] Cluster name: seatunnel-233929
2023-12-08 09:09:51,510 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1]

_____ _____ _
/ ___| |_ _| | |
\ `--. ___ __ _ | | _ _ _ __ _ __ ___ | |
`--. \ / _ \ / _` | | | | | | || '_ \ | '_ \ / _ \| |
/\__/ /| __/| (_| | | | | |_| || | | || | | || __/| |
\____/ \___| \__,_| \_/ \__,_||_| |_||_| |_| \___||_|


2023-12-08 09:09:51,511 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2023-12-08 09:09:51,511 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following:
- Change member config using Java API: config.setIntegrityCheckerEnabled(true);
- Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
- Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2023-12-08 09:09:51,513 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
- Change member config using Java API: config.getJetConfig().setEnabled(true)
- Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
- Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2023-12-08 09:09:51,867 INFO com.hazelcast.system.security - [localhost]:5801 [seatunnel-233929] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see security recommendations and the status of current config.
2023-12-08 09:09:51,919 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-233929] [5.1] Using TCP/IP discovery
2023-12-08 09:09:51,920 WARN com.hazelcast.cp.CPSubsystem - [localhost]:5801 [seatunnel-233929] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
十二月 08, 2023 9:09:52 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
十二月 08, 2023 9:09:52 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml
十二月 08, 2023 9:09:52 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
十二月 08, 2023 9:09:52 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml
2023-12-08 09:09:52,071 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
2023-12-08 09:09:52,072 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] Created new BusWork : 1217639236
2023-12-08 09:09:52,082 WARN org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - failed send heartbeat to resource manager, will retry later. this address: [localhost]:5801
2023-12-08 09:09:52,084 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1]
***********************************************
CoordinatorService Thread Pool Status
***********************************************
activeCount : 0
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 0
completedTaskCount : 0
taskCount : 0
***********************************************

2023-12-08 09:09:52,121 INFO com.hazelcast.internal.diagnostics.Diagnostics - [localhost]:5801 [seatunnel-233929] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-12-08 09:09:52,128 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5801 is STARTING
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5802 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5817 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5819 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5823 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5804 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5821 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5813 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5825 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5815 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5808 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5806 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5829 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5810 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5827 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5822 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5820 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5814 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5812 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5826 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5805 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5807 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5818 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5816 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5830 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5809 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5811 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5824 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5828 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5803 is added to the blacklist.
2023-12-08 09:09:53,162 INFO com.hazelcast.internal.cluster.ClusterService - [localhost]:5801 [seatunnel-233929] [5.1]

Members {size:1, ver:1} [
Member [localhost]:5801 - 87c80e4a-80c5-4b82-b577-cb0649025b5e this
]

2023-12-08 09:09:53,172 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5801 is STARTED
十二月 08, 2023 9:09:53 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
十二月 08, 2023 9:09:53 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast-client.yaml
2023-12-08 09:09:53,269 INFO com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel-233929] [5.1] Running with 2 response threads, dynamic=true
2023-12-08 09:09:53,279 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-12-08 09:09:53,279 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-12-08 09:09:53,285 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-233929] [5.1] Trying to connect to cluster: seatunnel-233929
2023-12-08 09:09:53,286 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-233929] [5.1] Trying to connect to [localhost]:5801
2023-12-08 09:09:53,299 INFO com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [localhost]:5801 [seatunnel-233929] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55059, qualifier=null, endpoint=[127.0.0.1]:55059, remoteUuid=535a4a1b-58b7-40a4-8909-218d69fedfe5, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 535a4a1b-58b7-40a4-8909-218d69fedfe5, client name: hz.client_1, client version: 5.1
2023-12-08 09:09:53,302 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-12-08 09:09:53,302 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-233929] [5.1] Authenticated with server [localhost]:5801:87c80e4a-80c5-4b82-b577-cb0649025b5e, server version: 5.1, local address: /127.0.0.1:55059
2023-12-08 09:09:53,303 INFO com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel-233929] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-12-08 09:09:53,308 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel-233929] [5.1]

Members [1] {
Member [localhost]:5801 - 87c80e4a-80c5-4b82-b577-cb0649025b5e
}

2023-12-08 09:09:53,329 INFO com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2023-12-08 09:09:53,453 INFO org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[]
2023-12-08 09:09:53,466 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: ../config/v2.batch.config.template
2023-12-08 09:09:53,504 INFO org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
2023-12-08 09:09:53,540 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
"env" : {
"execution.parallelism" : 2,
"job.mode" : "BATCH",
"checkpoint.interval" : 10000
},
"source" : [
{
"schema" : {
"fields" : {
"name" : "string",
"age" : "int"
}
},
"row.num" : 16,
"parallelism" : 2,
"result_table_name" : "fake",
"plugin_name" : "FakeSource"
}
],
"sink" : [
{
"plugin_name" : "Console"
}
]
}

2023-12-08 09:09:53,560 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,561 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,564 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel
2023-12-08 09:09:53,568 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
2023-12-08 09:09:53,568 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Console at: file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar
2023-12-08 09:09:53,571 INFO org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
2023-12-08 09:09:53,571 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,571 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,571 WARN org.apache.seatunnel.engine.core.parse.ConfigParserUtil - This configuration is not recommended. A source/transform(FakeSource) is configured with 'result_table_name' option value of 'fake', but subsequent transform/sink(Console) is not configured with 'source_table_name' option.
2023-12-08 09:09:53,572 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources.
2023-12-08 09:09:53,572 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,581 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel
2023-12-08 09:09:53,585 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
2023-12-08 09:09:53,586 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='FakeSource'} from classpath
2023-12-08 09:09:53,603 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all transforms.
2023-12-08 09:09:53,603 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sinks.
2023-12-08 09:09:53,603 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,607 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,636 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Start submit job, job id: 785312093409640449, with plugin jar [file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar, file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar]
2023-12-08 09:09:53,640 WARN org.apache.seatunnel.engine.server.SeaTunnelServer - This is master node, waiting the coordinator service init finished
2023-12-08 09:09:54,086 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1] This node become a new active master node, begin init coordinator service
2023-12-08 09:09:54,114 INFO com.hazelcast.internal.partition.impl.PartitionStateManager - [localhost]:5801 [seatunnel-233929] [5.1] Initializing cluster partition table arrangement...
2023-12-08 09:09:54,141 WARN org.apache.seatunnel.engine.server.SeaTunnelServer - This is master node, waiting the coordinator service init finished
2023-12-08 09:09:54,646 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - Init ResourceManager
2023-12-08 09:09:54,646 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker...
2023-12-08 09:09:54,646 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker live nodes: [[localhost]:5801]
2023-12-08 09:09:54,648 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - registerWorker: {[localhost]:5801=WorkerProfile(address=[localhost]:5801, profile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=1029177344}}, unassignedResource=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=1029177344}}, assignedSlots=[], unassignedSlots=[])}
2023-12-08 09:09:54,662 INFO org.apache.seatunnel.engine.server.master.JobMaster - Init JobMaster for Job SeaTunnel_Job (785312093409640449)
2023-12-08 09:09:54,662 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (785312093409640449) needed jar urls [file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar, file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar]
2023-12-08 09:09:54,874 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-12-08 09:09:54,976 INFO org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage - Path /tmp/seatunnel/checkpoint_snapshot/785312093409640449 is not a directory
2023-12-08 09:09:54,976 INFO org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage - No checkpoint found for job, job id is: 785312093409640449
2023-12-08 09:09:54,978 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] is in state CREATED when init state future
2023-12-08 09:09:54,979 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] is in state CREATED when init state future
2023-12-08 09:09:54,979 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] is in state CREATED when init state future
2023-12-08 09:09:54,981 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Submit job finished, job id: 785312093409640449, job name: SeaTunnel
2023-12-08 09:09:54,982 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job Job SeaTunnel_Job (785312093409640449) waiting for scheduler finished
2023-12-08 09:09:54,984 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - Job SeaTunnel_Job (785312093409640449) turn from state CREATED to SCHEDULED.
2023-12-08 09:09:54,987 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] turn from state CREATED to SCHEDULED.
2023-12-08 09:09:54,989 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] state from CREATED to SCHEDULED
2023-12-08 09:09:54,990 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] turn from state CREATED to SCHEDULED.
2023-12-08 09:09:54,994 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] state from CREATED to SCHEDULED
2023-12-08 09:09:54,994 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] turn from state CREATED to SCHEDULED.
2023-12-08 09:09:54,995 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] state from CREATED to SCHEDULED
2023-12-08 09:09:54,996 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] turn from state CREATED to SCHEDULED.
2023-12-08 09:09:55,003 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] turn from state SCHEDULED to DEPLOYING.
2023-12-08 09:09:55,005 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] state from SCHEDULED to DEPLOYING
2023-12-08 09:09:55,006 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] turn from state SCHEDULED to DEPLOYING.
2023-12-08 09:09:55,006 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] state from SCHEDULED to DEPLOYING
2023-12-08 09:09:55,007 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] turn from state SCHEDULED to DEPLOYING.
2023-12-08 09:09:55,008 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] state from SCHEDULED to DEPLOYING
2023-12-08 09:09:55,009 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] turn from state SCHEDULED to DEPLOYING.
2023-12-08 09:09:55,009 WARN org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run
2023-12-08 09:09:55,016 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] received deploying task executionId [785312099889840129]
2023-12-08 09:09:55,025 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] received deploying task executionId [785312099889971201]
2023-12-08 09:09:55,025 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] received deploying task executionId [785312099889905665]
2023-12-08 09:09:55,034 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1}, executionId [785312099889840129]
2023-12-08 09:09:55,035 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1} init success
2023-12-08 09:09:55,040 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1} success
2023-12-08 09:09:55,040 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] state from DEPLOYING to RUNNING
2023-12-08 09:09:55,042 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: pipeline-1 [Source[0]-FakeSource-fake]
2023-12-08 09:09:55,042 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] turn from state DEPLOYING to RUNNING.
2023-12-08 09:09:55,047 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}, executionId [785312099889971201]
2023-12-08 09:09:55,048 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001} init success
2023-12-08 09:09:55,048 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}, executionId [785312099889905665]
2023-12-08 09:09:55,049 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001} success
2023-12-08 09:09:55,050 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] state from DEPLOYING to RUNNING
2023-12-08 09:09:55,051 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000} init success
2023-12-08 09:09:55,052 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] turn from state DEPLOYING to RUNNING.
2023-12-08 09:09:55,052 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000} success
2023-12-08 09:09:55,053 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] state from DEPLOYING to RUNNING
2023-12-08 09:09:55,054 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] turn from state DEPLOYING to RUNNING.
2023-12-08 09:09:55,054 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] turn from state DEPLOYING to RUNNING.
2023-12-08 09:09:55,054 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 0
2023-12-08 09:09:55,054 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 1
2023-12-08 09:09:55,060 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - Job SeaTunnel_Job (785312093409640449) turn from state SCHEDULED to RUNNING.
2023-12-08 09:09:55,061 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (785312093409640449) scheduler finished
2023-12-08 09:09:55,064 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 0
2023-12-08 09:09:55,064 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 1
2023-12-08 09:09:55,068 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - output rowType: name<STRING>, age<INT>
2023-12-08 09:09:55,068 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - output rowType: name<STRING>, age<INT>
2023-12-08 09:09:55,166 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}, taskID=40001, index=1}
2023-12-08 09:09:55,166 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0}
2023-12-08 09:09:55,245 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received enough reader, starting enumerator...
2023-12-08 09:09:55,245 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Starting to calculate splits.
2023-12-08 09:09:55,246 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Assigned [FakeSourceSplit(splitId=1, rowNum=16), FakeSourceSplit(splitId=0, rowNum=16)] to 2 readers.
2023-12-08 09:09:55,246 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Calculated splits successfully, the size of splits is 2.
2023-12-08 09:09:55,247 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Assigning splits to readers 0 [FakeSourceSplit(splitId=0, rowNum=16)]
2023-12-08 09:09:55,251 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Assigning splits to readers 1 [FakeSourceSplit(splitId=1, rowNum=16)]
2023-12-08 09:09:55,269 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceReader - 16 rows of data have been generated in split(0). Generation time: 1701997795267
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceReader - Closed the bounded fake source
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceReader - 16 rows of data have been generated in split(1). Generation time: 1701997795266
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceReader - Closed the bounded fake source
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=1: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : kkTzr, 348993738
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=1: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : cHRlJ, 141341247
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=2: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : JoqKS, 949667288
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=2: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : wuRPB, 739004063
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=3: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : vzyWZ, 1185127730
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=3: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : MzsIj, 255990586
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=4: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : pKInq, 1307779433
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=4: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : JmaUB, 255860364
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=5: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : MYeRA, 1997823969
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=5: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : uVILC, 995215612
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=6: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : DlVbk, 731828921
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=7: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : inYyz, 1847828742
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=6: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : cyjxD, 1832606505
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=7: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : eMzXN, 802135106
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=8: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : kPUMq, 1916092538
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=8: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : aySgR, 1299495336
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=9: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : vFyQU, 1895773363
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=9: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : pjYqn, 1385074915
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=10: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : dQmgk, 1234357286
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=10: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : zASIv, 1497863961
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=11: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : zQuTA, 903385081
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=11: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : IVXOt, 870847193
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=12: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : jrLTl, 705955925
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=13: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : YflRj, 293817756
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=12: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : DbvZU, 1486490099
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=14: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : SeaoD, 1066109912
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=13: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : VsbMy, 910565833
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=15: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : vkEhB, 403795863
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=14: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : fegPV, 165600914
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=16: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : rXrEp, 1087983282
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=15: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : DokOA, 1281308771
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=16: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : VjDeD, 1091404164
2023-12-08 09:09:55,356 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 9223372036854775807
2023-12-08 09:09:55,411 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - pending checkpoint(9223372036854775807/1@785312093409640449) notify finished!
2023-12-08 09:09:55,411 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start notify checkpoint completed, checkpoint:org.apache.seatunnel.engine.server.checkpoint.CompletedCheckpoint@4d1c18a2
2023-12-08 09:09:55,416 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator completed.
2023-12-08 09:09:55,417 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_785312093409640449_1 state from null to FINISHED
2023-12-08 09:09:55,453 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1}
2023-12-08 09:09:55,453 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] Task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1} complete with state FINISHED
2023-12-08 09:09:55,454 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1] Received task end from execution TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1}, state FINISHED
2023-12-08 09:09:55,455 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] turn to end state FINISHED.
2023-12-08 09:09:55,455 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] end with state FINISHED
2023-12-08 09:09:55,464 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 50001, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}
2023-12-08 09:09:55,464 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 50000, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}
2023-12-08 09:09:56,271 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 40001, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}
2023-12-08 09:09:56,271 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 40000, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}
2023-12-08 09:09:56,272 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] Task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000} complete with state FINISHED
2023-12-08 09:09:56,272 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] Task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001} complete with state FINISHED
2023-12-08 09:09:56,272 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1] Received task end from execution TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}, state FINISHED
2023-12-08 09:09:56,272 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1] Received task end from execution TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}, state FINISHED
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] turn to end state FINISHED.
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] turn to end state FINISHED.
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] end with state FINISHED
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] end with state FINISHED
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] end with state FINISHED
2023-12-08 09:09:56,310 INFO org.apache.seatunnel.engine.server.master.JobMaster - release the pipeline Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] resource
2023-12-08 09:09:56,311 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 785312093409640449, slot: SlotProfile{worker=[localhost]:5801, slotID=1, ownerJobID=785312093409640449, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='fa7bad5a-cd2c-4d97-8368-88027b3b49b6'}
2023-12-08 09:09:56,311 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 785312093409640449, slot: SlotProfile{worker=[localhost]:5801, slotID=2, ownerJobID=785312093409640449, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='fa7bad5a-cd2c-4d97-8368-88027b3b49b6'}
2023-12-08 09:09:56,312 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 785312093409640449, slot: SlotProfile{worker=[localhost]:5801, slotID=3, ownerJobID=785312093409640449, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='fa7bad5a-cd2c-4d97-8368-88027b3b49b6'}
2023-12-08 09:09:56,314 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] turn to end state FINISHED.
2023-12-08 09:09:56,315 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - Job SeaTunnel_Job (785312093409640449) end with state FINISHED
2023-12-08 09:09:56,328 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Job (785312093409640449) end with state FINISHED
2023-12-08 09:09:56,353 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -
***********************************************
Job Statistic Information
***********************************************
Start Time : 2023-12-08 09:09:53
End Time : 2023-12-08 09:09:56
Total Time(s) : 2
Total Read Count : 32
Total Write Count : 32
Total Failed Count : 0
***********************************************

2023-12-08 09:09:56,354 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-12-08 09:09:56,357 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801 [seatunnel-233929] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55059, qualifier=null, endpoint=[127.0.0.1]:55059, remoteUuid=535a4a1b-58b7-40a4-8909-218d69fedfe5, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2023-12-08 09:09:56,357 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-233929] [5.1] Removed connection to endpoint: [localhost]:5801:87c80e4a-80c5-4b82-b577-cb0649025b5e, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:55059->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-12-08 09:09:56.351, lastWriteTime=2023-12-08 09:09:56.328, closedTime=2023-12-08 09:09:56.355, connected server version=5.1}
2023-12-08 09:09:56,358 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-12-08 09:09:56,358 INFO com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801 [seatunnel-233929] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55059, qualifier=null, endpoint=[127.0.0.1]:55059, remoteUuid=535a4a1b-58b7-40a4-8909-218d69fedfe5, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=535a4a1b-58b7-40a4-8909-218d69fedfe5, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1701997793296, latest clientAttributes=lastStatisticsCollectionTime=1701997793329,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1701997793287,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=24435175424,os.freePhysicalMemorySize=50645405696,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=65536,os.openFileDescriptorCount=58,os.processCpuTime=9670000000,os.systemLoadAverage=2.31,os.totalPhysicalMemorySize=540422402048,os.totalSwapSpaceSize=0,runtime.availableProcessors=104,runtime.freeMemory=959249984,runtime.maxMemory=1029177344,runtime.totalMemory=1029177344,runtime.uptime=2770,runtime.usedMemory=69927360, labels=[]}
2023-12-08 09:09:56,360 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-12-08 09:09:56,360 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-12-08 09:09:56,360 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5801 is SHUTTING_DOWN
2023-12-08 09:09:56,363 INFO com.hazelcast.internal.partition.impl.MigrationManager - [localhost]:5801 [seatunnel-233929] [5.1] Shutdown request of Member [localhost]:5801 - 87c80e4a-80c5-4b82-b577-cb0649025b5e this is handled
2023-12-08 09:09:56,367 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-233929] [5.1] Shutting down connection manager...
2023-12-08 09:09:56,369 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-233929] [5.1] Shutting down node engine...
2023-12-08 09:09:58,473 INFO com.hazelcast.instance.impl.NodeExtension - [localhost]:5801 [seatunnel-233929] [5.1] Destroying node NodeExtension.
2023-12-08 09:09:58,474 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-233929] [5.1] Hazelcast Shutdown is completed in 2111 ms.
2023-12-08 09:09:58,474 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5801 is SHUTDOWN
2023-12-08 09:09:58,474 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
2023-12-08 09:09:58,474 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......
2023-12-08 09:09:58,475 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal
[opensource@bigdata02 bin]$

🍓 启动服务

🍒启动服务并且后台运行,用于web端连接: nohup sh seatunnel-cluster.sh 2>&1 &
在这里插入图片描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

[opensource@bigdata02 bin]$ ls
install-plugin.sh seatunnel.sh start-seatunnel-flink-15-connector-v2.sh start-seatunnel-spark-3-connector-v2.sh
seatunnel-cluster.sh start-seatunnel-flink-13-connector-v2.sh start-seatunnel-spark-2-connector-v2.sh stop-seatunnel-cluster.sh
[opensource@bigdata02 bin]$
[opensource@bigdata02 bin]$ nohup sh seatunnel-cluster.sh 2>&1 &
[1] 93523
[opensource@bigdata02 bin]$ nohup: 忽略输入并把输出追加到"nohup.out"

[opensource@bigdata02 bin]$ ls
install-plugin.sh seatunnel-cluster.sh start-seatunnel-flink-13-connector-v2.sh start-seatunnel-spark-2-connector-v2.sh stop-seatunnel-cluster.sh
nohup.out seatunnel.sh start-seatunnel-flink-15-connector-v2.sh start-seatunnel-spark-3-connector-v2.sh
[opensource@bigdata02 bin]$ tail -f nohup.out
十二月 08, 2023 9:30:58 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml
十二月 08, 2023 9:30:59 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
十二月 08, 2023 9:30:59 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml
十二月 08, 2023 9:30:59 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
十二月 08, 2023 9:30:59 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml
^C
[opensource@bigdata02 bin]$ tail -f ../logs/seatunnel-engine-server.log
Member [localhost]:5801 - 9db58520-2335-441c-8999-ad78123f6509 this
]

2023-12-08 09:30:59,930 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel] [5.1] [localhost]:5801 is STARTED
2023-12-08 09:31:01,865 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel] [5.1] This node become a new active master node, begin init coordinator service
2023-12-08 09:31:02,061 INFO com.hazelcast.internal.partition.impl.PartitionStateManager - [localhost]:5801 [seatunnel] [5.1] Initializing cluster partition table arrangement...
2023-12-08 09:31:04,863 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - Init ResourceManager
2023-12-08 09:31:04,863 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker...
2023-12-08 09:31:04,864 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker live nodes: [[localhost]:5801]
2023-12-08 09:31:04,865 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - registerWorker: {[localhost]:5801=WorkerProfile(address=[localhost]:5801, profile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=2058354688}}, unassignedResource=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=2058354688}}, assignedSlots=[], unassignedSlots=[])}
2023-12-08 09:31:59,863 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel] [5.1]
***********************************************
CoordinatorService Thread Pool Status
***********************************************
activeCount : 0
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 0
completedTaskCount : 0
taskCount : 0
***********************************************

2023-12-08 09:31:59,867 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel] [5.1]
***********************************************
Job info detail
***********************************************
createdJobCount : 0
scheduledJobCount : 0
runningJobCount : 0
failingJobCount : 0
failedJobCount : 0
cancellingJobCount : 0
canceledJobCount : 0
finishedJobCount : 0
restartingJobCount : 0
suspendedJobCount : 0
reconcilingJobCount : 0
***********************************************

🍊 手动安装

  • 不想全部安装,修改配置vi config/plugin_config (具体保留哪些就不清楚了,只能说官方提供的安装包不好,无法直接启动,必须要先安装插件,但是这么多插件下载又很慢、也没必要)
    在这里插入图片描述

  • 无法在线安装,只能手动了
    maven库下载地址
    在这里插入图片描述

    对于自动下载下来的jar包,已经全部打包上传到CSDN,大家可以免费下载,直接放到 connectors/seatunnel
    在这里插入图片描述

    下载地址
    在这里插入图片描述

    CSDN资源大小不允许超过1000M,这里采用.tar.xz格式(解压会有些慢,小十分钟吧)

  • 三种压缩命令:```bash
    tar -cf seatunnel_connectors_2.3.3.tar seatunnel
    tar -czf seatunnel_connectors_2.3.3.tar.gz seatunnel
    tar -cJf seatunnel_connectors_2.3.3.tar.xz seatunnel

    1
    2
    3
    4
    5

    - 对应解压命令:```bash
    tar -xf seatunnel_connectors_2.3.3.tar
    tar -xzf seatunnel_connectors_2.3.3.tar.gz
    tar -xJf seatunnel_connectors_2.3.3.tar.xz

    在这里插入图片描述

🍑三、Web界面部署


🍊 上传解压web部署包: tar xf apache-seatunnel-web-1.0.0-bin.tar.gz
在这里插入图片描述

🍊 初始化数据库

🍓初始化SQL脚本 script/seatunnel_server_mysql.sql
在这里插入图片描述

在这里插入图片描述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
SET FOREIGN_KEY_CHECKS = 0;

-- ----------------------------
-- Table structure for role
-- ----------------------------
DROP TABLE IF EXISTS `role`;
CREATE TABLE `role` (
`id` int(20) NOT NULL AUTO_INCREMENT,
`type` int(2) NOT NULL,
`role_name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
`description` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,
`create_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------
-- Records of role
-- ----------------------------
INSERT INTO `role`(`type`,`role_name`,`description`) values (0, 'ADMIN_ROLE', 'Admin User');
INSERT INTO `role`(`type`,`role_name`,`description`) values (1, 'NORMAL_ROLE', 'Normal User');

-- ----------------------------
-- Table structure for role_user_relation
-- ----------------------------
DROP TABLE IF EXISTS `role_user_relation`;
CREATE TABLE `role_user_relation` (
`id` int(20) NOT NULL AUTO_INCREMENT,
`role_id` int(20) NOT NULL,
`user_id` int(20) NOT NULL,
`create_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for t_st_datasource
-- ----------------------------
DROP TABLE IF EXISTS `t_st_datasource`;
CREATE TABLE `t_st_datasource` (
`id` bigint(20) NOT NULL,
`datasource_name` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`plugin_name` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`plugin_version` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT '1.0.0',
`datasource_config` varchar(1023) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`description` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(11) NOT NULL,
`update_user_id` int(11) NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `t_st_datasource_datasource_name_uindex`(`datasource_name`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for t_st_job_definition
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_definition`;
CREATE TABLE `t_st_job_definition` (
`id` bigint(20) NOT NULL,
`name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`description` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`job_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(11) NOT NULL,
`update_user_id` int(11) NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `name`(`name`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for t_st_job_instance
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_instance`;
CREATE TABLE `t_st_job_instance` (
`id` bigint(20) NOT NULL,
`job_define_id` bigint(20) NOT NULL,
`job_status` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`job_config` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`engine_name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`engine_version` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`job_engine_id` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(20) NOT NULL,
`update_user_id` int(20) NULL DEFAULT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
`end_time` timestamp(3) NULL DEFAULT NULL,
`job_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for t_st_job_instance_history
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_instance_history`;
CREATE TABLE `t_st_job_instance_history` (
`id` bigint(20) NOT NULL,
`dag` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for t_st_job_line
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_line`;
CREATE TABLE `t_st_job_line` (
`id` bigint(20) NOT NULL,
`version_id` bigint(20) NOT NULL,
`input_plugin_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`target_plugin_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE,
INDEX `job_line_version_index`(`version_id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for t_st_job_metrics
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_metrics`;
CREATE TABLE `t_st_job_metrics` (
`id` bigint(20) NOT NULL,
`job_instance_id` bigint(20) NOT NULL,
`pipeline_id` int(20) NOT NULL,
`read_row_count` bigint(20) NOT NULL,
`write_row_count` bigint(20) NOT NULL,
`source_table_names` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`sink_table_names` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`read_qps` bigint(20) NULL DEFAULT NULL,
`write_qps` bigint(20) NULL DEFAULT NULL,
`record_delay` bigint(20) NULL DEFAULT NULL,
`status` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(20) NOT NULL,
`update_user_id` int(20) NULL DEFAULT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for t_st_job_task
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_task`;
CREATE TABLE `t_st_job_task` (
`id` bigint(20) NOT NULL,
`version_id` bigint(20) NOT NULL,
`plugin_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`config` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL,
`transform_options` varchar(5000) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`output_schema` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL,
`connector_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`datasource_id` bigint(20) NULL DEFAULT NULL,
`datasource_option` varchar(5000) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`select_table_fields` varchar(5000) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`scene_mode` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE,
INDEX `job_task_plugin_id_index`(`plugin_id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for t_st_job_version
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_version`;
CREATE TABLE `t_st_job_version` (
`id` bigint(20) NOT NULL,
`job_id` bigint(20) NOT NULL,
`name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`job_mode` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`env` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL,
`engine_name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`engine_version` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`create_user_id` int(11) NOT NULL,
`update_user_id` int(11) NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for t_st_virtual_table
-- ----------------------------
DROP TABLE IF EXISTS `t_st_virtual_table`;
CREATE TABLE `t_st_virtual_table` (
`id` bigint(20) NOT NULL,
`datasource_id` bigint(20) NOT NULL,
`virtual_database_name` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`virtual_table_name` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`table_fields` varchar(1023) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`virtual_table_config` varchar(1023) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`description` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(11) NOT NULL,
`update_user_id` int(11) NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for user
-- ----------------------------
DROP TABLE IF EXISTS `user`;
CREATE TABLE `user` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`username` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
`password` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
`status` tinyint(4) NOT NULL,
`type` tinyint(4) NOT NULL,
`create_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 2 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------
-- Table structure for user_login_log
-- ----------------------------
DROP TABLE IF EXISTS `user_login_log`;
CREATE TABLE `user_login_log` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`token` mediumtext CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
`token_status` tinyint(1) NOT NULL,
`create_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 106 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------
-- Records of user_login_log
-- ----------------------------

INSERT INTO `user`(`username`,`password`,`status`,`type`) values ('admin', '7f97da8846fed829bb8d1fd9f8030f3b', 0, 0);

SET FOREIGN_KEY_CHECKS = 1;

🍊 配置文件修改

🍓配置数据库信息: vi conf/application.yml
在这里插入图片描述

🍓复制引擎服务中配置文件到web配置目录下面

1
2
[opensource@bigdata02 app]$ cp apache-seatunnel-2.3.3/config/hazelcast-client.yaml apache-seatunnel-web-1.0.0-bin/conf/
[opensource@bigdata02 app]$ cp apache-seatunnel-2.3.3/connectors/plugin-mapping.properties apache-seatunnel-web-1.0.0-bin/conf/

在这里插入图片描述

🍊 配置下载数据源jar包

🍓选择的mysql作为元数据库,对应的驱动包 mysql-connector-java-8.0.25.jar 放到 libs
在这里插入图片描述

🍓下载其它数据源资源包,首先下载源码,然后执行下载脚本 download_datasource.sh (本地安装过git)

  • 🍒 源码地址
    在这里插入图片描述

  • 🍒修改 download_datasource.sh 相关配置
    在这里插入图片描述

    在这里插入图片描述

  • 🍒执行 download_datasource.sh
    在这里插入图片描述

  • 🍒开始下载jar包
    在这里插入图片描述

  • 🍒下载完成
    在这里插入图片描述

  • 🍒上传到 libs 目录
    在这里插入图片描述

🍊 启动服务

1
sh bin/seatunnel-backend-daemon.sh start

在这里插入图片描述

🍑四、web页面一览


🍊登录

🍓登录地址: http://xxxx:8801/ui/#/login ; 用户密码 admin/admin
在这里插入图片描述

在这里插入图片描述

🍊数据源

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

🍊任务

在这里插入图片描述

选不到数据源…
在这里插入图片描述

在这里插入图片描述

🍓尝试创建整库同步
在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

🍊用户管理

在这里插入图片描述

🍊虚拟表

在这里插入图片描述

🍑五、其它


🍊问题记录

🍓服务启动失败: Application run failed org.yaml.snakeyaml.scanner.ScannerException: while scanning an anchor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
14:29:08.032 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
org.yaml.snakeyaml.scanner.ScannerException: while scanning an anchor
in 'reader', line 30, column 15:
password: &R7#J*vDVWEuT48m
^
unexpected character found *(42)
in 'reader', line 30, column 20:
password: &R7#J*vDVWEuT48m
^

at org.yaml.snakeyaml.scanner.ScannerImpl.scanAnchor(ScannerImpl.java:1507)
at org.yaml.snakeyaml.scanner.ScannerImpl.fetchAnchor(ScannerImpl.java:958)
at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:389)
at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:248)
at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingValue.produce(ParserImpl.java:665)
at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:165)
at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:59)
at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:45)
at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:140)
at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:119)
at org.yaml.snakeyaml.composer.Composer.composeScalarN

🍓解决办法,密码 &R7#J*vDVWEuT48m 增加双引号
在这里插入图片描述

在这里插入图片描述

🍊总结

  • 🍓 apache-seatunnel-web-1.0.0-bin 还处在开发早期阶段,并不成熟,功能也不完善

  • 🍓安装seatunnel的目的,就是想参考下web界面,寻求针对表的拖拽处理(减少sql编写),目前看效果不理想,也只是一个简单的输入、输出表字段映射,因此暂告一段落
    在这里插入图片描述

评论

:D 一言句子获取中...