UDF(Java): permission denied at HDFS

2019-09-15 09:20发布

I write an hive-UDTF to resolve ip address via loading a .dat file on HDFS,but meet an error:

java.io.FileNotFoundException: hdfs:/THE_IP_ADDRESS:9000/tmp/ip_20170204.dat (Permission denied) 

But actually, both the dfs directory /tmp and the .data file have full access: 777, and I cannot modify the config to disable dfs permission.

The line in my UDTF to read the file:

IP.load("hdfs://THE_IP_ADDRESS:9000/tmp/ip_20170204.dat");

and the static method .load():

public static void load(String filename) {
        ipFile = new File(filename);
        load();
        if (enableFileWatch) {
            watch();
        }
    }


private static void load() {
    lastModifyTime = ipFile.lastModified();
    FileInputStream fin = null;
    lock.lock();
    try {
        dataBuffer = ByteBuffer.allocate(Long.valueOf(ipFile.length()).intValue());
        fin = new FileInputStream(ipFile);
        int readBytesLength;
        byte[] chunk = new byte[4096];
        while (fin.available() > 0) {
            readBytesLength = fin.read(chunk);
            dataBuffer.put(chunk, 0, readBytesLength);
        }
        dataBuffer.position(0);
        int indexLength = dataBuffer.getInt();
        byte[] indexBytes = new byte[indexLength];
        dataBuffer.get(indexBytes, 0, indexLength - 4);
        indexBuffer = ByteBuffer.wrap(indexBytes);
        indexBuffer.order(ByteOrder.LITTLE_ENDIAN);
        offset = indexLength;

        int loop = 0;
        while (loop++ < 256) {
            index[loop - 1] = indexBuffer.getInt();
        }
        indexBuffer.order(ByteOrder.BIG_ENDIAN);
    } catch (IOException ioe) {
        ioe.printStackTrace();
    } finally {
        try {
            if (fin != null) {
                fin.close();
            }
        } catch (IOException e){
            e.printStackTrace();
        }
        lock.unlock();
    }
}

标签: java hive hdfs udf
0条回答
登录 后发表回答