scala – sbt程序集任务在添加一些依赖项后运行缓慢

Have you read the 07001. It specifically suggests that you might change the cacheUnzip and cacheOutput settings. I would give it a try.

cacheUnzip是一个优化功能,但cacheOutput不是. cacheOutput的目的是在源未更改时获取相同的jar.对于某些人来说,重要的是输出罐不会不必要地改变.需要注意的是,它正在检查所有* .class文件的SHA-1哈希值.所以自述文件说：

If there are a large number of class files,this could take a long time

据我所知,合并策略的解压缩和应用需要大约一两分钟,但SHA-1的检查似乎需要永远.这是关闭输出缓存的assembly.sbt：

import AssemblyKeys._ // put this at the top of the file

assemblySettings

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>  {
    case PathList("javax","servlet",xs @ _*)         => MergeStrategy.first
    case PathList("org","apache","commons",xs @ _*) => MergeStrategy.first // commons-beanutils-core-1.8.0.jar vs commons-beanutils-1.7.0.jar
    case PathList("com","esotericsoftware","minlog",xs @ _*) => MergeStrategy.first // kryo-2.21.jar vs minlog-1.2.jar
    case "about.html"                                  => MergeStrategy.rename
    case x => old(x)
  }
}

assemblyCacheOutput in assembly := false

清洁后58秒内完成组装,第二次运行无需清洁需要15秒.虽然有些跑步也需要200秒.

看一下源代码,我可能会优化cacheOutput,但是现在关闭它应该会使组件更快.

编辑：

我已根据这个问题添加了#96 Performance degradation when adding library dependencies,并在sbt-assembly 0.10.1为sbt 0.13添加了一些修复.

sbt-assembly 0.10.1 avoids content hashing of the unzipped items of the dependent library jars. It also skips jar caching done by sbt,since sbt-assembly is already caching the output.

The changes make assembly task run more consistently. Using deps-heavy spark as sample project,assembly task was run 15 times after a small source change. sbt-assembly 0.10.0 took 19+/-157 seconds (mostly within 20 secs,but going 150+ secs 26% of the runs). On the other hand,sbt-assembly 0.10.1 took 16+/-1 seconds.