Flink开发-生成Jar(Maven依赖处理)

发表于 | 分类于
Flink开发-生成Jar(Maven依赖处理)

打包设置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
<packaging>jar</packaging>

<build>
  <plugins>
    <plugin>
      <groupId>net.alchim31.maven</groupId>
      <artifactId>scala-maven-plugin</artifactId>
      <version>3.4.6</version>
      <executions>
        <execution>
          <goals>
            <goal>compile</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-assembly-plugin</artifactId>
      <version>3.0.0</version>
      <configuration>
        <archive>
          <manifest>
            <mainClass>com.xhkjedu.TCS</mainClass>
          </manifest>
        </archive>
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
      </configuration>
      <executions>
        <execution>
          <id>make-assembly</id>
          <phase>package</phase>
          <goals>
            <goal>single</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

运行报错

Unable to instantiate java compiler

原因

flink服务器上的 flink-table-blink_2.11-1.12.0.jar 将calcite等依赖打入，因此用户的jar包中需要排除calcite相关依赖。

用户项目中可使用maven helper搜索calcite相关依赖，将其provided，并将这些依赖放入flink lib目录下，重启flink服务即可

常见的包含calcite的依赖有：

  • hive-exec
  • flink-table-planner
  • flink-table-planner-blink
  • calcite-core

Flink部署的jar在项目中是首先要排除provided的

示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
<dependency>
  <groupId>org.apache.hive</groupId>
  <artifactId>hive-exec</artifactId>
  <version>${hive.version}</version>
  <scope>provided</scope>
</dependency>

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  <version>${flink.version}</version>
  <scope>provided</scope>
</dependency>

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-table-planner-blink_2.12</artifactId>
  <version>${flink.version}</version>
  <scope>provided</scope>
</dependency>

Jar冲突

Flink在运行的时候会先加载安装目录下的lib中的所有jar，要保证项目重复的jar都标记为<scope>provided</scope>。