logo
down
shadow

Is there any existing API to read and write to HDFS from JAVA


Is there any existing API to read and write to HDFS from JAVA

By : Ritesh Ranjan SIngh
Date : November 21 2020, 07:38 AM
this one helps. Is there any existing API that exists to read/write from HDFS, along with best practices of how to implement it . , Below is the code snippet
code :
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Progressable;



String Dest = "/user/pkumar/test.xml";

 Configuration conf = new Configuration();
       FileSystem fs = FileSystem.get(URI.create(Dest),conf);
        Path path = new Path(Dest);

    if(!fs.exists(path)){
        OutputStream out = fs.create(path, new Progressable(){
            public void progress(){
                System.out.print(".");
            }
        });
        System.out.println();
        IOUtils.copyBytes(connect, out, 4096, true);
    }
<dependencies>
<dependency>
     <groupId>org.apache.hadoop</groupId>
     <artifactId>hadoop-common</artifactId>
     <version>2.3.0-cdh5.1.3</version>
</dependency>
<dependency>
     <groupId>org.apache.hadoop</groupId>
     <artifactId>hadoop-client</artifactId>
     <version>2.3.0-cdh5.1.3</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.3.2</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.3.2</version>
</dependency>

</dependencies>


Share : facebook icon twitter icon
Java with HDFS file read/write

Java with HDFS file read/write


By : appleLikeOrange
Date : March 29 2020, 07:55 AM
will help you I figured out the solution for this error. And looks like I was using the wrong port. I was using the port number as I see on HUE URL ( misleaded from different sources).
If I chose the port number as defined for the configuration "NameNode Service RPC Port" OR "dfs.namenode.servicerpc-address" on the name node from Cloudera manager, it works fine.
Java Read and Write Spark Vector's to Hdfs

Java Read and Write Spark Vector's to Hdfs


By : prvn
Date : March 29 2020, 07:55 AM
help you fix your problem Spark directly supports reading Hadoop SequenceFiles. You would do something like:
code :
JavaSparkContext sc = new JavaSparkContext(conf);
JavaPairRDD<LongWritable, Vector> input = 
    sc.sequenceFile(fileName, LongWritable.class, Vector.class);
JavaRDD<Vector> out = input.map(new Function<Tuple2<LongWritable, Vector>, Vector>() {

    @Override
    public Vector call(Tuple2<LongWritable, Vector> tuple) throws Exception {
        return tuple._2();
    }
});
Is it possible to read and write Parquet using Java without a dependency on Hadoop and HDFS?

Is it possible to read and write Parquet using Java without a dependency on Hadoop and HDFS?


By : Daniel Mondria
Date : March 29 2020, 07:55 AM
wish help you to fix your issue You can write parquet format out side hadoop cluster using java Parquet Client API.
Here is a sample code in java which writes parquet format to local disk.
code :
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
import org.apache.hadoop.fs.Path;
import org.apache.parquet.avro.AvroSchemaConverter;
import org.apache.parquet.avro.AvroWriteSupport;
import org.apache.parquet.hadoop.ParquetWriter;
import org.apache.parquet.hadoop.metadata.CompressionCodecName;
import org.apache.parquet.schema.MessageType;

public class Test {
    void test() throws IOException {
        final String schemaLocation = "/tmp/avro_format.json";
        final Schema avroSchema = new Schema.Parser().parse(new File(schemaLocation));
        final MessageType parquetSchema = new AvroSchemaConverter().convert(avroSchema);
        final WriteSupport<Pojo> writeSupport = new AvroWriteSupport(parquetSchema, avroSchema);
        final String parquetFile = "/tmp/parquet/data.parquet";
        final Path path = new Path(parquetFile);
        ParquetWriter<GenericRecord> parquetWriter = new ParquetWriter(path, writeSupport, CompressionCodecName.SNAPPY, BLOCK_SIZE, PAGE_SIZE);
        final GenericRecord record = new GenericData.Record(avroSchema);
        record.put("id", 1);
        record.put("age", 10);
        record.put("name", "ABC");
        record.put("place", "BCD");
        parquetWriter.write(record);
        parquetWriter.close();
    }
}
{
   "type":"record",
   "name":"Pojo",
   "namespace":"com.xx.test",
   "fields":[
      {
         "name":"id",
         "type":[
            "int",
            "null"
         ]
      },
      {
         "name":"age",
         "type":[
            "int",
            "null"
         ]
      },
      {
         "name":"name",
         "type":[
            "string",
            "null"
         ]
      },
      {
         "name":"place",
         "type":[
            "string",
            "null"
         ]
      }
   ]
}
Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 FAIL

Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 FAIL


By : Sujan Bhujel
Date : March 29 2020, 07:55 AM
should help you out Thank for everyone. I lost 1 week for this bug. Just by mistake version.
1) Hadoop 3.x.x dont support hive 1.x.x
read and write data simultaneously in an existing file in java

read and write data simultaneously in an existing file in java


By : Ali Hellani
Date : March 29 2020, 07:55 AM
I hope this helps . you typically have to read the whole file and then write the whole file, typically, for safety write to a temporary file, then when done delete the original and rename the temp file to the original name.
Related Posts Related Posts :
  • Failed to find provider info for urbanairship.provider
  • Android ListView Adapter Crash issue/Duplicates
  • Displaying Splash screen only once on user preference
  • Loop List<WebElement> doesn't stop despite no more WebElements
  • Check if pattern exists in a String
  • Swing - Key Problems (Arrow Keys, Tab, etc.) in GUIs with TextComponents
  • Why the solution for binary search tree validity is not working?
  • Double encrypting 2048 RSA?
  • create junit test for runnable class
  • Unable to read and sort information from a text file in Java
  • My device isn't getting detected in android studio (driver issue)
  • Representing a Graph in Java
  • How to XML mapping using jxls
  • Quick Sort stackoverflow error for large arrays
  • Parsing Apache Tika XML Output returns Unknown Tag
  • java simple UI loop, can't find bug
  • how to use greater than and less than in a single if statement in java
  • Error:(27, 13) Failed to resolve: com.antonyt.infiniteviewpager:library:1.0.0
  • Gradle dependencies on source set of different projects
  • Session Scope in Restful Spring Services
  • Java: Change a value in an overridden method of a different class to a value that was calculated in the class
  • Event notification if SWT Combo display list is visible
  • Hibernate WrongClassException for Custom Discriminators
  • User define enum element method in Java? /Hearthstone Text Game/
  • Java Multi-dimensional Array Average
  • Doubly Linked List delete last
  • First time Fragment, and I got "Application has stopped" when I try open it with fragmentTransaction
  • How to model two one-to-one relationships between the same two tables in Hibernate?
  • IntelliJ - include pom.xml dependencies in lib directory
  • Tomcat 7 : Filtering lifecycle events
  • JSONException: Names must be strings
  • Build Spark JavaRDD List from DropResult objects
  • Include css file in richlet with zk
  • Jackson CSV's WRAP_AS_ARRAY
  • Priority for a web request in Java web applications
  • Is there a better way to format this timestamp to ISO8601?
  • Issue with Spring Scope
  • How to declare and use an array of dictionaries from Python in Java and filter them?
  • Wrong weekdays from Calendar.getDisplayName()
  • ClassNotFoundException when creating new ClassPathXmlApplicationContext
  • Java Method Call from Thread
  • Remove background noise from image to make text more clear for OCR
  • PostgreSQL query always return true
  • How to connect FX controller with main app
  • Is there a difference between these two code snippets, and if so what?
  • When two interfaces have conflicting return types, why does one method become default?
  • Updating database throws errors in To Do List app
  • Where does the url come from?
  • Dagger 2 issue overriding single provides annotated method from a module in a library which app uses
  • Better way then If(x instanceof y) in Java?
  • how to debug maven enunciate plugin tomcat web service
  • Spring Context Initialization fails when upgrading to version 4.2.3
  • spring boot cannot connect to rabbitmq
  • How to Run Unit Test in Parallel Across All Modules Using Maven?
  • Getting the next Value in a ArrayList(Not working)
  • How to wait for a MouseListener mouse press?
  • Java RMI: Blocking for void method return?
  • Get Table Properties out of Hive using Java API
  • Read CSV file and write to another CSV - ArrayIndexOutOfBoundsException and pattern difficuties
  • Set LineChart(javafx) style to a child element
  • shadow
    Privacy Policy - Terms - Contact Us © animezone.co