java – FileChannel ByteBuffer和Hashing Files

我在 java中构建了一个文件哈希方法,它接受文件路径文件名的输入字符串表示,然后计算该文件的哈希值.散列可以是任何本机支持的java散列算法,例如MD2到SHA-512.

我试图找出最后一滴性能,因为这个方法是我正在研究的项目的一个组成部分.我被建议尝试使用FileChannel而不是常规的FileInputStream.

我原来的方法：

/**
     * Gets Hash of file.
     * 
     * @param file String path + filename of file to get hash.
     * @param hashAlgo Hash algorithm to use. <br/>
     *     Supported algorithms are: <br/>
     *     MD2,MD5 <br/>
     *     SHA-1 <br/>
     *     SHA-256,SHA-384,SHA-512
     * @return String value of hash. (Variable length dependent on hash algorithm used)
     * @throws IOException If file is invalid.
     * @throws HashTypeException If no supported or valid hash algorithm was found.
     */
    public String getHash(String file,String hashAlgo) throws IOException,HashTypeException {
        StringBuffer hexString = null;
        try {
            MessageDigest md = MessageDigest.getInstance(validateHashType(hashAlgo));
            FileInputStream fis = new FileInputStream(file);

            byte[] dataBytes = new byte[1024];

            int nread = 0;
            while ((nread = fis.read(dataBytes)) != -1) {
                md.update(dataBytes,nread);
            }
            fis.close();
            byte[] mdbytes = md.digest();

            hexString = new StringBuffer();
            for (int i = 0; i < mdbytes.length; i++) {
                hexString.append(Integer.toHexString((0xFF & mdbytes[i])));
            }

            return hexString.toString();

        } catch (NoSuchAlgorithmException | HashTypeException e) {
            throw new HashTypeException("Unsuppored Hash Algorithm.",e);
        }
    }

重构方法：

/**
     * Gets Hash of file.
     * 
     * @param file String path + filename of file to get hash.
     * @param hashAlgo Hash algorithm to use. <br/>
     *     Supported algorithms are: <br/>
     *     MD2,SHA-512
     * @return String value of hash. (Variable length dependent on hash algorithm used)
     * @throws IOException If file is invalid.
     * @throws HashTypeException If no supported or valid hash algorithm was found.
     */
    public String getHash(String fileStr,HasherException {

        File file = new File(fileStr);

        MessageDigest md = null;
        FileInputStream fis = null;
        FileChannel fc = null;
        ByteBuffer bbf = null;
        StringBuilder hexString = null;

        try {
            md = MessageDigest.getInstance(hashAlgo);
            fis = new FileInputStream(file);
            fc = fis.getChannel();
            bbf = ByteBuffer.allocate(1024); // allocation in bytes

            int bytes;

            while ((bytes = fc.read(bbf)) != -1) {
                md.update(bbf.array(),bytes);
            }

            fc.close();
            fis.close();

            byte[] mdbytes = md.digest();

            hexString = new StringBuilder();

            for (int i = 0; i < mdbytes.length; i++) {
                hexString.append(Integer.toHexString((0xFF & mdbytes[i])));
            }

            return hexString.toString();

        } catch (NoSuchAlgorithmException e) {
            throw new HasherException("Unsupported Hash Algorithm.",e);
        }
    }

两者都返回正确的哈希值,但重构的方法似乎只对小文件合作.当我传入一个大文件时,它完全窒息而我无法弄清楚原因.我是NIO的新手所以请指教.

编辑：忘了提到我正在通过它投掷SHA-512进行测试.

更新：使用我现在的方法更新.

/**
     * Gets Hash of file.
     * 
     * @param file String path + filename of file to get hash.
     * @param hashAlgo Hash algorithm to use. <br/>
     *     Supported algorithms are: <br/>
     *     MD2,HasherException {

        File file = new File(fileStr);

        MessageDigest md = null;
        FileInputStream fis = null;
        FileChannel fc = null;
        ByteBuffer bbf = null;
        StringBuilder hexString = null;

        try {
            md = MessageDigest.getInstance(hashAlgo);
            fis = new FileInputStream(file);
            fc = fis.getChannel();
            bbf = ByteBuffer.allocateDirect(8192); // allocation in bytes - 1024,2048,4096,8192

            int b;

            b = fc.read(bbf);

            while ((b != -1) && (b != 0)) {
                bbf.flip();

                byte[] bytes = new byte[b];
                bbf.get(bytes);

                md.update(bytes,b);

                bbf.clear();
                b = fc.read(bbf);
            }

            fis.close();

            byte[] mdbytes = md.digest();

            hexString = new StringBuilder();

            for (int i = 0; i < mdbytes.length; i++) {
                hexString.append(Integer.toHexString((0xFF & mdbytes[i])));
            }

            return hexString.toString();

        } catch (NoSuchAlgorithmException e) {
            throw new HasherException("Unsupported Hash Algorithm.",e);
        }
    }

所以我尝试使用我的原始示例和我最新的更新示例,对2.92GB文件的MD5进行基准测试.当然,任何基准测试都是相对的,因为存在操作系统和磁盘缓存以及其他会导致重复读取相同文件的“魔法”……但这里有一些基准测试.我把每个方法加载起来并在将它编译为新鲜后将其关闭5次.基准测试取自最后一次(第5次),因为这将是该算法的“最热门”运行,以及任何“魔术”(在我的理论中无论如何).

Here's the benchmarks so far: 

    Original Method - 14.987909 (s) 
    Latest Method - 11.236802 (s)

散布相同的2.92GB文件所花费的时间减少了25.03％.非常好.

while (fc.read(bbf) != -1) {
    md.update(bbf.array(),bytes);
    bbf.clear();
}

If this stream has an associated channel then the channel is closed as well.

ByteBuffer.allocateDirect(1024);