If you want to push all of your production server system log files into HDFS use Flume
Apache Flume is a distributed system for collecting streaming data. It’s an Apache
project in incubator status, originally developed by Cloudera. It offers various levels of
reliability and transport delivery guarantees that can be tuned to your needs. It’s
highly customizable and supports a plugin architecture where you can add custom
data sources and data sinks.
Link:
If you need to automate the process by which files on remote servers are copied into HDFS use HDFS File slurper.
Feautures
- After a successful file copy you can either remove the source file, or have it moved into another directory.
- Destination files can be compressed as part of the write codec with any compression codec which extends
org.apache.hadoop.io.compress.CompressionCodec. - Capability to write "done" file after completion of copy
- Verify destination file post-copy with CRC32 checksum comparison with source
- Ignores hidden files (filenames that start with ".")
- Customizable destination via a script which can be called for every source file. Or alternatively let the utility know a single destination directory
- Customizable pre-processing of file prior to transfer via script and all files are copied into that location.
- A daemon mode which is compatible with
inittabrespawn - Multi-threaded data transfer
Link:
If you want to automate peredioc tasksfor downloading content from an HTTP server into HDFS use:
Oozie is a workflow scheduler system to manage Apache Hadoop jobs.
Link:
http://oozie.apache.org/If you want to import relational data using MapReduce use DBInputFormat class
You can do the same using scoop
http://sqoop.apache.org/
$ bin/run.sh org.apache.hadoop.hbase.mapreduce.Export \
stocks_example \ - table name
output - directory
Or specific column family:
$ bin/run.sh org.apache.hadoop.hbase.mapreduce.Export \
-D hbase.mapreduce.scan.column.family=details \
-D mapred.output.compress=true \
-D mapred.output.compression.codec=\
org.apache.hadoop.io.compress.SnappyCodec \
stocks_example output
The Export class writes the HBase output in the SequenceFile format, where the HBase
rowkey is stored in the SequenceFile record key using org.apache.hadoop.hbase
.io.ImmutableBytesWritable, and the HBase value is stored in the SequenceFile record
value using org.apache.hadoop.hbase.client.Result.
Now its time to move to HDFS ( example of Stock records move):
import static com.manning.hip.ch2.HBaseWriteAvroStock.*;
public class HBaseExportedStockReader {
public static void main(String... args) throws IOException {
read(new Path(args[0]));
}
public static void read(Path inputPath) throws IOException {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
SequenceFile.Reader reader =new SequenceFile.Reader(fs, inputPath, conf);
HBaseScanAvroStock.AvroStockReader stockReader =
new HBaseScanAvroStock.AvroStockReader();
try {
ImmutableBytesWritable key = new ImmutableBytesWritable();
Result value = new Result();
while (reader.next(key, value)) {
Stock stock = stockReader.decode(value.getValue(
STOCK_DETAILS_COLUMN_FAMILY_AS_BYTES,
STOCK_COLUMN_QUALIFIER_AS_BYTES));
System.out.println(new String(key.get()) + ": " +ToStringBuilder.reflectionToString(stock,ToStringStyle.SIMPLE_STYLE));
}
} finally {
reader.close();
}
}
}
