Version: v1.1.3 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jun 22, 2018 License: MIT Imports: 16 Imported by: 70



Package rpc implements some of the lower-level functionality required to communicate with the namenode and datanodes.



This section is empty.


View Source
var ErrEndOfBlock = errors.New("The amount of data to be written is more than is left in the block.")
View Source
var ErrInvalidSeqno = errors.New("Invalid ack sequence number")


This section is empty.


type BlockReader

type BlockReader struct {
	// contains filtered or unexported fields

BlockReader implements io.ReadCloser, for reading a block. It abstracts over reading from multiple datanodes, in order to be robust to connection failures, timeouts, and other shenanigans.

func NewBlockReader

func NewBlockReader(block *hdfs.LocatedBlockProto, offset int64, clientName string) *BlockReader

NewBlockReader returns a new BlockReader, given the block information and security token from the namenode. It will connect (lazily) to one of the provided datanode locations based on which datanodes have seen failures.

func (*BlockReader) Close

func (br *BlockReader) Close() error

Close implements io.Closer.

func (*BlockReader) Read

func (br *BlockReader) Read(b []byte) (int, error)

Read implements io.Reader.

In the case that a failure (such as a disconnect) occurs while reading, the BlockReader will failover to another datanode and continue reading transparently. In the case that all the datanodes fail, the error from the most recent attempt will be returned.

Any datanode failures are recorded in a global cache, so subsequent reads, even reads for different blocks, will prioritize them lower.

type BlockWriter added in v1.0.0

type BlockWriter struct {
	// contains filtered or unexported fields

BlockWriter implements io.WriteCloser for writing a block to a datanode. Given a block location, it handles pipeline construction and failures, including communicating with the namenode if need be.

func NewBlockWriter added in v1.0.0

func NewBlockWriter(block *hdfs.LocatedBlockProto, namenode *NamenodeConnection, blockSize int64) *BlockWriter

NewBlockWriter returns a BlockWriter for the given block. It will lazily set up a replication pipeline, and connect to the "best" datanode based on any previously seen failures.

func (*BlockWriter) Close added in v1.0.0

func (bw *BlockWriter) Close() error

Close implements io.Closer. It flushes any unwritten packets out to the datanode, and sends a final packet indicating the end of the block.

func (*BlockWriter) Flush added in v1.1.1

func (bw *BlockWriter) Flush() error

Flush flushes any unwritten packets out to the datanode.

func (*BlockWriter) Write added in v1.0.0

func (bw *BlockWriter) Write(b []byte) (int, error)

Write implements io.Writer.

Unlike BlockReader, BlockWriter currently has no ability to recover from write failures (timeouts, datanode failure, etc). Once it returns an error from Write or Close, it may be in an invalid state.

This will hopefully be fixed in a future release.

type ChecksumReader

type ChecksumReader struct {
	// contains filtered or unexported fields

ChecksumReader provides an interface for reading the "MD5CRC32" checksums of individual blocks. It abstracts over reading from multiple datanodes, in order to be robust to failures.

func NewChecksumReader

func NewChecksumReader(block *hdfs.LocatedBlockProto) *ChecksumReader

NewChecksumReader creates a new ChecksumReader for the given block.

func (*ChecksumReader) ReadChecksum

func (cr *ChecksumReader) ReadChecksum() ([]byte, error)

ReadChecksum returns the checksum of the block.

type NamenodeConnection

type NamenodeConnection struct {
	// contains filtered or unexported fields

NamenodeConnection represents an open connection to a namenode.

func NewNamenodeConnection

func NewNamenodeConnection(address string, user string) (*NamenodeConnection, error)

NewNamenodeConnection creates a new connection to a namenode and performs an initial handshake.

You probably want to use hdfs.New instead, which provides a higher-level interface.

func NewNamenodeConnectionWithOptions added in v1.1.0

func NewNamenodeConnectionWithOptions(options NamenodeConnectionOptions) (*NamenodeConnection, error)

NewNamenodeConnectionWithOptions creates a new connection to a namenode with the given options and performs an initial handshake.

func WrapNamenodeConnection deprecated

func WrapNamenodeConnection(conn net.Conn, user string) (*NamenodeConnection, error)

WrapNamenodeConnection wraps an existing net.Conn to a Namenode, and preforms an initial handshake.

Deprecated: use the higher-level hdfs.New or NewNamenodeConnection instead.

func (*NamenodeConnection) ClientName added in v1.0.0

func (c *NamenodeConnection) ClientName() string

ClientName provides a unique identifier for this client, which is required for various RPC calls. Confusingly, it's separate from clientID, which is used in the RPC header; to make things simpler, it reuses the random bytes from that, but adds a prefix to make it human-readable.

func (*NamenodeConnection) Close added in v1.0.0

func (c *NamenodeConnection) Close() error

Close terminates all underlying socket connections to remote server.

func (*NamenodeConnection) Execute

func (c *NamenodeConnection) Execute(method string, req proto.Message, resp proto.Message) error

Execute performs an rpc call. It does this by sending req over the wire and unmarshaling the result into resp.

type NamenodeConnectionOptions added in v1.1.0

type NamenodeConnectionOptions struct {
	Addresses []string
	User      string

NamenodeConnectionOptions represents the configurable options available for a NamenodeConnection.

type NamenodeError

type NamenodeError struct {
	Method    string
	Message   string
	Code      int
	Exception string

NamenodeError represents an interepreted error from the Namenode, including the error code and the java backtrace.

func (*NamenodeError) Desc

func (err *NamenodeError) Desc() string

Desc returns the long form of the error code, as defined in the RpcErrorCodeProto in RpcHeader.proto

func (*NamenodeError) Error

func (err *NamenodeError) Error() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL