org.hd.d.pg2k.svrCore.datasource
Class ExhibitDataTunnelSource

java.lang.Object
  org.hd.d.pg2k.svrCore.datasource.ExhibitDataTunnelSource

All Implemented Interfaces:

SimpleExhibitPipelineIF, BasicVarMgrInterface, SimpleVariablePipelineIF

Direct Known Subclasses:

ExhibitDataHTTPTunnelSource, SystemVariablesTest.LocalTunnelSource

public abstract class ExhibitDataTunnelSource
extends java.lang.Object
implements SimpleExhibitPipelineIF

Exhibit pipeline stage that fetches its data across a master/slave tunnel. Some of the visible types and values are used by the implementations of both ends of the tunnel (eg the tunnel servlet) to help implement the shared protocol.

This takes data from the server's own DataSourceBean by default.

FORCE_GC_AND_FINALISERS_AFTER_POLL_FAIL

private static final boolean FORCE_GC_AND_FINALISERS_AFTER_POLL_FAIL

If true, try GC and running finalizers after some connection failures. If we think that connection failure may have been caused by failure to release resources (elsewhere in the VM!) then try to force a clean-up from time to time. But, in case we are wrong, do not do this too often anyway.

See Also:

Constant Field Values

FAIL_RETRY_WAIT_MIN_MS

public static final int FAIL_RETRY_WAIT_MIN_MS

Minimum wait after failed tunnel call before retry (ms); strictly positive. A few tens of milliseconds to a second is probably best, to recover from transient I/O problems quickly, but without actually busy-waiting...

See Also:

Constant Field Values

FAIL_RETRY_WAIT_MAX_MS

public static final int FAIL_RETRY_WAIT_MAX_MS

Maximum wait after failed tunnel call before retry (ms). Several minutes is probably good, to cope with a crashed master server (and/or to wait for it to reboot), or a failed connection, but too long and our distributed variables will fail needlessly.

Most requests during the wait will be vetoed cleanly and quickly allowing timely service to the slave's clients.

We vary this limit slightly between slaves/instances to help avoid them colliding with one another especially if hitting this upper bound.

KEEP_SERVER_CONNECTION_ALIVE

protected static final boolean KEEP_SERVER_CONNECTION_ALIVE

If true, allow connection/resource sharing between calls if possible. We may also periodically check that we can still contact the master.

An open connection would probably improve TCP efficiency and reduce latency for an underlying HTTP[S] connection, for example. (We might allow HTTP/1.1 streaming and connection sharing to be done for us by the JDK runtime if this is true.)

See Also:

Constant Field Values

_protocolDebug

protected static final boolean _protocolDebug

If true, extra instrumentation for protocol debug; definitely not for release code.

See Also:

Constant Field Values

uniqueClientID

public final SimpleVariableValue uniqueClientID

Our unique client-side end-point identifier. The ID goes into all outgoing variable updates (and possibly other data) going upstream to the master to uniquely identify this client.

This is unique to each tunnel client end-point instance and is created afresh on each run, so a rebooted client gets a new ID.

Uses secure/good generator to try to ensure global uniqueness. (This may be expensive to compute; we might be able to defer this until first needed.)

logger

protected final SimpleLoggerIF logger

Our logger which falls back to System.out if servlet log not available; never null.

statsIDTS

private final StatsLogger.StatsConfig statsIDTS

The stats set to which we log general tunnel source stats. The unique codes are the constants TSNAME_XXX.

TSNAME_RPCREQUEST

public static final java.lang.String TSNAME_RPCREQUEST

General stats event name: RPC request.

See Also:

Constant Field Values

TSNAME_RPCIOEX

public static final java.lang.String TSNAME_RPCIOEX

General stats event name: RPC failure due to an IOException.

See Also:

Constant Field Values

TSNAMEPR_RPCTYPE

public static final java.lang.String TSNAMEPR_RPCTYPE

General stats event name prefix: RPC request packet type.

See Also:

Constant Field Values

TSNAME_SHORTREAD

public static final java.lang.String TSNAME_SHORTREAD

General stats event name: "short" raw data read, less than bulk-transfer block size..

See Also:

Constant Field Values

EMPTY_PAYLOAD

public static final byte[] EMPTY_PAYLOAD

Convenience value; (immutable) zero-length byte array for use as an empty packet payload. Used by local classes and TunnelServlet; package visible.

lastSuccessfulConnectionTime

private volatile long lastSuccessfulConnectionTime

Time when last successful connection was made, or 0 if no connection.. Declared volatile so that it can be accessed without a lock.

Set by _noteResult() after a good exchange with the master; never cleared and never null.

doNotTryMasterUntil

private volatile java.lang.Long doNotTryMasterUntil

Time before which we should not again try to contact the master; null if master is fine. Declared volatile so that it can be accessed without a lock.

Cleared to null by _noteResult() after a good exchange with the master; the wait to another attempt is approximately doubled after a failed exchange.

successiveFailureCount

private int successiveFailureCount

Successive failure count; reset to zero upon success. Private to _noteResult() and accessed under its private lock.

Cleared to zero on a successful packet exchange; incremented after each failure.

_nR_lock

private final java.lang.Object _nR_lock

Private lock for _noteResult.

NO_EVENT_VALUES

private static final EventVariableValue[] NO_EVENT_VALUES

Immutable empty result indicating no event values available.

MAX_AEP_DIFF_COMP_LEVEL

public static final CompressionLevel MAX_AEP_DIFF_COMP_LEVEL

Maximum level of compression in AEP "diff" RPC supported in this implementation, client or server side.

_REPAIR_RETRY_LEAD_MS

private final int _REPAIR_RETRY_LEAD_MS

How soon ahead of other callers we try to silently repair a broken connection, ms; non-negative.

MIN_AEP_RETENTION_MS

public static final int MIN_AEP_RETENTION_MS

Minimum time (ms) for which we will hold latest newly-created AEP response; strictly positive. A client that automatically retries after an interrupted IO response to some sort of AEP request indicating 'already in progress' should retry within this time to avoid starting its creation all over again.

Note that in dire straits this lower limit may be ignored.

Should typically be of the order of many minutes.

See Also:

Constant Field Values

BLOCK_RAW_FILE_CACHEING

private static final boolean BLOCK_RAW_FILE_CACHEING

If true then never pass through a request to cache content locally. Content may be cached anyway, but avoiding passing through a cache request here may help break request 'loops' and/or conserve master cache space for example.

See Also:

Constant Field Values

ExhibitDataTunnelSource

public ExhibitDataTunnelSource(SimpleLoggerIF logger)

Create an instance, passing in a (non-null) logger to use.

getLastSuccessfulConnectionTime

public long getLastSuccessfulConnectionTime()

Get time of last successful connection, or 0 if none.

isBroken

public boolean isBroken()

Returns true if we currently cannot talk to the master, false otherwise.

_noteResult

private void _noteResult(boolean success)

Routine to note success or failure of RPC call and adjust control variables. Has its own private lock to ensure atomic update of control variables.

Parameters:

success - call with this true immediately after successful RPC exchange with master; call with false otherwise

doNOOP

public void doNOOP(boolean unguarded)
            throws java.io.IOException

Does a NO-OP on the server. If this returns without throwing an exception, then the connection to the back-end (master) server is probably OK.

Parameters:

unguarded - if true, this ignores any recent connection problems and immediately tries to contact the master, else it behaves like other operations and is quickly vetoed while the master is failing

Throws:

java.io.IOException - in case of trouble communicating with master.

getStaticAttr

public ExhibitStaticAttr getStaticAttr(Name.ExhibitFull name)
                                throws java.io.IOException

Get the static attributes for a given exhibit. Returns null if the named exhibit does not exist.

Sends name as a UTF string and allows compression. (We use UTF since we know that the length is limited and the all-ASCII nature of valid names should yield one-byte-per-char encoding before compression.)

This will fail with a null name argument.

(Big secret: if this class is used behind, for example, ExhibitDataSimpleCache, this method will never be called because all such calls are answered direct from the cache.)

Specified by:

getStaticAttr in interface SimpleExhibitPipelineIF

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

getRawFile

public void getRawFile(java.nio.ByteBuffer buf,
                       Name.ExhibitFull exhibitName,
                       int position,
                       boolean dontCache)
                throws java.io.IOException

Get a chunk of the raw exhibit binary. The start position must be non-negative.

Sends name as a UTF string and allows compression of the request. (We use UTF since we know that the length is limited and the all-ASCII nature of valid names should yield one-byte-per-char encoding before compression.)

We send the start and (computed) afterEnd arguments as int values.

We return the data as-is.

Specified by:

getRawFile in interface SimpleExhibitPipelineIF

Parameters:

buf - the buffer into which to read the data; must be non-null, in put()able state, and with remaining capacity of at least the requested number of bytes

exhibitName - the full name of the exhibit to read from; never null and must be syntactically valid

position - position/index of first byte in exhibitFile to read; non-negative

dontCache - if true, this is a hint not to attempt to cache this or displace anything from extant caches for this data as it may for example be precaching or random activity; by default callers should leave this false to allow cacheing

Throws:

java.lang.IllegalArgumentException - for blatantly invalid name, or non-positive length

java.io.IOException - for requests that cannot be fulfilled because of I/O restrictions or problems, such as link failure or an upper bound on the length of a request

getAllExhibitImmutableData

public AllExhibitImmutableData getAllExhibitImmutableData(long oldStamp)
                                                   throws java.io.IOException

Gets all static exhibit data if its timestamp is not that specified. If the time specified is negative the object will be returned unconditionally.

If no exhibits are currently installed a default set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldStamp matches that that we would have been returned) null is returned.

Specified by:

getAllExhibitImmutableData in interface SimpleExhibitPipelineIF

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

getAllExhibitProperties

public AllExhibitProperties getAllExhibitProperties(long oldHash)
                                             throws java.io.IOException

Gets set of all exhibit properties if its hash is not that specified. If the hash specified is negative the object will be returned unconditionally.

If no exhibits are currently installed then a default set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldHash matches that that would have been returned) then null is returned.

Because the AEP drives the entire Gallery, we always let the request go over the tunnel even if recent calls have failed. For example, we don't want mirrors to remain out of sync for too long.

This also typically takes a lot of bandwidth and CPU/wallclock time, so is optimised and stripped down to a minimum, and relies on the AEP deserialisation for serious error checking.

It is hoped that using the streamed form will overlap I/O and CPU work and thus get the new AEP into the client quicker and smoother.

We attempt to be good citizens and note most RPC successes/failures to help maintain the tunnel's view of the master's status.

(Because satisfying these calls may take a long time, the upstream server may veto concurrent queries that return anything other than null.)

As a further attempt to minimise time and bandwidth, we may try the "diff" version of this call, falling back to the normal version if this fails.

Specified by:

getAllExhibitProperties in interface SimpleExhibitPipelineIF

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

getAllExhibitProperties

public AllExhibitProperties getAllExhibitProperties(AllExhibitProperties oldAEP,
                                                    boolean allowAutoRecovery)
                                             throws java.io.IOException

Gets set of all exhibit properties if not that specified, attempting minimise data transferred across the tunnel. This is a tunnel-specific optimisation to minimise bandwidth. For some clients this will save money and time.

If the AEP specified is null then the remote AEP will be fetched and returned unconditionally.

If no exhibits are currently installed then a default set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldHash matches that that would have been returned) then null is returned.

This is an attempted optimisation of the getAllExhibitProperties(long oldHash), returning a diff or extra-highly-compressed AEP representation.

This may not be supported at all by the remote end of the connection, or may fail for lack of resources, etc, so we can automatically fall back to the usual call in case of difficulty.

(We do not record failures as connection/tunnel problems, though we do note successful calls in favour of the connection status.)

Parameters:

oldAEP - current latest AEP held by the caller

allowAutoRecovery - if true then allow fallback to generic AEP fetch method

Throws:

java.io.IOException

longSer

public static byte[] longSer(long value)

Helper method to serialise a single free-standing long value. The long value is serialised just as DataOutputStream would, as 8 bytes with the high-byte first.

intSer

public static byte[] intSer(int value)

Helper method to serialise a single free-standing int value. The int value is serialised just as DataOutputStream would, as 4 bytes with the high-byte first.

getGenProps

public GenProps getGenProps(long oldStamp)
                     throws java.io.IOException

Gets the general properties as a GenProps object if its timestamp is not that specified. If the time specified is negative the object will be returned unconditionally.

If no props are currently installed/available a default set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldStamp matches that that we would have been returned) null is returned.

On the wire this is (outbound) a 8-byte timestamp and by return should be an empty payload (corresponding to null) or a serialised GenProps object (uncompressed for now).

Because GenProps are so important to the functioning of the entire Gallery, we always let the request go over the tunnel even if recent calls have failed.

Specified by:

getGenProps in interface SimpleExhibitPipelineIF

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

getGenSecProps

public java.util.Properties getGenSecProps(long oldStamp)
                                    throws java.io.IOException

Gets the security properties as a Properties object if its timestamp is not that specified. If the time specified is negative the object will be returned unconditionally.

If no props are currently installed/available a default set with a zero timestamp is returned.

If the caller's copy appears to be up-to-date (eg the oldStamp matches that that would have been returned) null is returned.

Specified by:

getGenSecProps in interface SimpleExhibitPipelineIF

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

getThumbnails

public ExhibitThumbnails getThumbnails(Name.ExhibitFull name,
                                       boolean create)
                                throws java.io.IOException

Gets the thumbnails for an exhibit. A data source is at liberty to refuse to compute thumbnails in which case it may return null, else it returns a non-null value which may include the `could-not-compute' value to indicate that a thumbnail/sample cannot be made for this exhibit and no attempt need be made in future.

Because it is important for a new mirror to gather thumbnails ASAP to show to a user, we make this call immune to being blocked when the tunnel connection appears poor.

Specified by:

getThumbnails in interface SimpleExhibitPipelineIF

Parameters:

create - if true, and no thumbnail yet exists, try to create one if possible, else only return an existing one

Throws:

java.io.IOException - if the operation cannot be completed due to I/O restrictions or failure

setVariable

public void setVariable(SimpleVariableValue newValue)
                 throws java.io.IOException,
                        java.lang.UnsupportedOperationException

Set variable. Only non-local values are propagated upstream; others are explicitly rejected with an UnsupportedOperationException.

Specified by:

setVariable in interface BasicVarMgrInterface

Throws:

java.io.IOException - in case of I/O problems

java.lang.UnsupportedOperationException - when attempting to set locals

adjustGlobalMapForSet

private SimpleVariableValue adjustGlobalMapForSet(SimpleVariableValue svv)

Adjust globalMap for outgoing (upstream) set operation. This expects the value passed to be:

• Non-null
• Non-local
• Read/write
• To have an empty globalMap or one with exactly one entry whose key is the system ID for this system; if the map is empty then this system is put in the global map

If the argument is in the correct form, it is returned unchanged.

If the argument is illegal, ie not suitable to send upstream, then an exception is thrown to veto the send.

Throws:

java.lang.IllegalArgumentException - if an invalid argument is passed, eg unable to be converted to be sent upstream because it has an invalid globalMap

setVariables

public int setVariables(SimpleVariableValue[] newValues)
                 throws java.io.IOException,
                        java.lang.UnsupportedOperationException

Set variables; must not be null or contain nulls. Only non-local variables are sent upstream; locals are silently discarded.

Variables not defined in SystemVariables are rejected.

Duplicates may be discarded or rejected, or sent as is in which case it is undefined in which order they are applied.

We try to set all values that we can for

Format on the wire:

• Outbound: serialised SimpleVariableValue[]
• Inbound: number of values set (int)

Specified by:

setVariables in interface BasicVarMgrInterface

Returns:

the number of variable values set (the length of the array); never negative, never more than the number passed in

Throws:

java.io.IOException - in case of I/O problems

java.lang.UnsupportedOperationException

getVariable

public SimpleVariableValue getVariable(SimpleVariableDefinition var)
                                throws java.io.IOException,
                                       java.lang.UnsupportedOperationException

Get variable, or returns null if no such non-local variable. We attempt only to fetch non-local variables; return null for others (except for the system ID).

When asked for the the system ID, we always answer the question locally.

Note that the local system ID can be retrieved whether or not the master server is responding.

Format on the wire is:

• Outbound: variable name in UTF format
• Inbound: serialised form of SimpleVariableValue (or empty for null)

Specified by:

getVariable in interface BasicVarMgrInterface

Parameters:

var - definition of variable to fetch; never null

Throws:

java.io.IOException - in case of I/O error

java.lang.UnsupportedOperationException - if a local variable other than the system ID is requested

getVariables

public SimpleVariableValue[] getVariables(long changedSince)
                                   throws java.io.IOException

Fetch variable values from the master. We fetch values from the master (from which we eliminate any illegal values such as local variables or values which are not locally valid) and insert our unique local system ID if the request stamp is -1 or older than our creation.

Note that the local system ID can be retrieved whether or not the master server is responding providing the stamp is -1, though in that case it might be the only value returned. The caller should periodically use -1 to ensure that they have a full set of global variables, eg in case the server rebooted.

Format on the wire:

• Outbound: serialised timestamp
• Inbound: SimpleVariableValue[]

Specified by:

getVariables in interface BasicVarMgrInterface

Throws:

java.io.IOException

getEventValue

public EventVariableValue getEventValue(SimpleVariableDefinition def,
                                        EventPeriod intervalSelector,
                                        boolean current)

Get the current partial, or previous full, event set at the specified interval; never null. This is a simplified interface to return either the current event set that is being collected, or the previous completed set.

The current set is the most timely, but may not contain enough data to be meaningful if the new interval has just started.

The previous set is complete and thus most likely to have enough samples to be useful, but is not completely current.

Implemented in terms of the more general call in the hope that batched calls for several values will be more common and more efficient. This also reduces the number of distinct RPC calls that we have to implement.

Specified by:

getEventValue in interface BasicVarMgrInterface

Parameters:

def - event definition (must be for an event); never null

intervalSelector - one of EVENT_INTERVAL_SELECTOR_xxx values

current - if true the current event set is returned, else the previous complete set is returned

Returns:

requested event set; may be empty but never null if requested set not available

getEventValues

public EventVariableValue[] getEventValues(SimpleVariableDefinition def,
                                           EventPeriod intervalSelector,
                                           long intervalNumber,
                                           java.util.BitSet whichValues)

Get the specified global event sets for the specified intervals; never null. This allows retrieval of zero or more event sets for the specified interval size.

Requests for more than SystemVariables.EVENT_SAMPLES_RETAINED in the past (or for the future!) cannot be satisfied and data will not be returned for them.

Usually not more than SystemVariables.EVENT_SAMPLES_RETAINED samples will be returned in response to any one request as a safety measure.

We do various small optimisations to reduce fruitless traffic over what may be a slow and/or expensive connection:

• This will reject requests will huge sets that go back to far so as to avoid huge RPC arguments and since the request is unlikely to be satisfiable. The upstream implementation may be even more restrictive.
• This ignores requests too far in the future or the past.
• Trim leading empty slots from request sent over the tunnel.

The responses from the server may be large and slow, so we allow streaming to give better incremental CPU/resource consumption and better throughput.

Returns an empty result if asked for a 'local' value.

Returns an empty result if the tunnel is currently broken or in case of other non-permanent error.

TODO: Optimise RPC call by allowing for sparse requests with different request/response format.

Specified by:

getEventValues in interface BasicVarMgrInterface

Parameters:

def - event definition (must be for an event); never null

intervalSelector - one of EVENT_INTERVAL_SELECTOR_xxx values

intervalNumber - a time (as from System.currentTimeMillis()) which identifies the first interval for which data is potentially required; if too far in the past or future then possibly no data will be available, zero is used to access the "all" bucket

whichValues - each true bit represents a slot for which data is required, bit 0 indicating data from the slot within which firstIntervalTime is located, bit 1 the previous slot, etc; null is treated as the common case equivalent to just bit 0 set

Returns:

as many of the requested values as available, at least long enough to return all the available values, with [0] corresponding to bit 0 in the BitSet; may contain nulls or be zero-length (eg in case of error) but is never null

syncVariables

public void syncVariables(boolean force)
                   throws java.io.IOException

Synchronise variables with upstream values. Pushes updated values upstream to the source, calls sync on the source if called with the "force" argument true, and then retrieves changed values from upstream.

When called with force==true, this acts like a full "memory barrier", flushing all write-cached items downstream immediately and afterwards getting the value of all upstream values with getVariables(-1), but may be expensive in terms of CPU or bandwidth, so use sparingly.

When called with force=false, this returns immediately and the operation is not propagated across the tunnel.

In any case, it is rarely the right thing for a casual user to vall this as it may be very expensive.

Specified by:

syncVariables in interface SimpleVariablePipelineIF

Parameters:

force - if true, this will force a full write flush, a full sync upstream, then full read with getVariables(-1), to get the effect of a full "barrier"; otherwise, do nothing

Throws:

ExhibitDataTunnelSource.TunnelBusyIOException - if one is received from upstream

java.io.IOException

getProperties

public java.util.Properties getProperties(SimpleExhibitPipelineIF.PropsKey key,
                                          long versionID)
                                   throws java.io.IOException

Get requested Properties selected by key and versionID. Fetches a Properties set unconditionally (versionID == -1) else if the versionID presented is not current.

Specified by:

getProperties in interface SimpleExhibitPipelineIF

Parameters:

key - selector (with possible embedded sub-key) for desired properties set; never null

versionID - if -1 then map is always returned if available, else must be non-negative and null is returned if the versionID presented matches that of the current version (ie if the caller has presumably got the up-to-date version); may be a timestamp or a hash or other value, and by convention is zero only for an empty properties set

Returns:

null, or Properties map guaranteed to contain only String keys and values

Throws:

java.io.IOException

getStratum

public Stratum getStratum()
                   throws java.io.IOException

Returns the (incremented) upstream stratum adjusted to include transit time; never null. Returns Stratum.UNKNOWN if the upstream stratum is unknown or the upstream instance is already at the maximum stratum.

Specified by:

getStratum in interface SimpleExhibitPipelineIF

Throws:

java.io.IOException

_getStratumUpstreamName

protected java.lang.String _getStratumUpstreamName()

Return short unique name of upstream peer/server suitable for Stratum; never null but can be "" for 'unknown'. Override this in implementations that know the upstream name.

poll

public void poll(GenProps gp)

Poll periodically. We can use this to attempt to repair ailing connections, etc, as well as poll for asynchronous messages being sent to us by the master.

However, our general policy is not to force traffic if we need not.

This can be overridden by a derived class, though it is suggested that this be called with super.poll(gp) if so.

Specified by:

poll in interface SimpleExhibitPipelineIF

destroy

public void destroy()

May attempt to free up outbound connections and/or prevent new ones.

Specified by:

destroy in interface SimpleExhibitPipelineIF

doRPCRaw

protected abstract ExhibitDataTunnelSource.RawPacket doRPCRaw(ExhibitDataTunnelSource.RawPacket packetOut)
                                                       throws java.io.IOException

Make an RPC call over the underlying medium with the given outgoing packet; never null. This collects the response packet and will object if it sees any IOException or if the packets come back with the wrong op code.

This can be locked on the instance lock to serialise RPCs, if the tunnel can only usefully handle one call at once.

Must be implemented by the deriving class to suit its transmission medium.

Parameters:

packetOut - request packet; never null

Returns:

response packet; never null

Throws:

java.io.IOException - in case of I/O difficulties

doRPCRawWithStreamResponse

protected java.io.InputStream doRPCRawWithStreamResponse(ExhibitDataTunnelSource.RawPacket packetOut,
                                                         boolean allowBigReadTimeout)
                                                  throws java.io.IOException

Optimised RPC call with the given outgoing packet and returning packet body as an InputStream; null if an empty stream. This is an optimisation for performance-critical cases only, and foregoes some error checking/handling for speed (and thus the caller should ensure that it performs integrity checks). There is an onus on the streaming-related code to behave safely even if fed bogus/corrupt data, and it must be able to safely/easily undo any work done if the message is found to be bogus as late as the final byte(s).

This streams the content of the response packet and will object if it sees any IOException or if the packets come back with the wrong op code.

A terminating trailer byte may or may not be visible on the returned stream thus allowing the implementation to be as efficient as possible.

This may return after all the input data has been collected, or while some or all is still to come, and thus the returned stream may fail and throw an exception.

The first element of the result is the length of the response data (not including any non-data trailer bytes from the packet even if present) but may be null if this length is not known when the packet header is seen, eg because the packet body was compressed.

This may be implemented/overridden by the deriving class to suit its transmission medium, and as an optimisation to reduce copying and allow streaming, ie starting to process the input before it is all received.

The data stream is always of uncompressed data, regardless of whether the data was sent compressed on the wire, ie this routine will correctly decompress data on the fly as/when needed.

The caller must close the stream promptly to release resources such as file handles and non-Java memory.

Parameters:

packetOut - request packet; never null

allowBigReadTimeout - TODO

Returns:

response length and data stream; never null

Throws:

java.io.IOException - in case of I/O difficulties

doRPC

protected ExhibitDataTunnelSource.RawPacket doRPC(ExhibitDataTunnelSource.RawPacket packetOut)
                                           throws java.io.IOException

Make an RPC call over HTTP[S] with the given outgoing packet. This provides error recovery, back-off, stats, etc, and serves as a wrapper for the medium-specific doRPCRaw().

Called by all the public data-pipeline methods to transport information over the tunnel.

Parameters:

packetOut - request packet; never null

Returns:

response packet; never null

Throws:

java.io.IOException - in case of I/O difficulties

doRPCUnguarded

protected ExhibitDataTunnelSource.RawPacket doRPCUnguarded(ExhibitDataTunnelSource.RawPacket packetOut)
                                                    throws java.io.IOException

Just like doRPC() but does not back off in face of previous failures; never null. Can be used for calls that must attempt to contact the master anyway, or are probing for it to be alive.

Most callers should use the normal doRPC().

Parameters:

packetOut - never null

Returns:

response to RPC call; never null

Throws:

java.io.IOException

handleInboundRPC

public static ExhibitDataTunnelSource.RawPacket handleInboundRPC(SimpleExhibitPipelineIF source,
                                                                 ExhibitDataTunnelSource.RawPacket reqPacket,
                                                                 java.lang.String clientAddr,
                                                                 ExhibitDataTunnelSource.HIRPCCache cache,
                                                                 SimpleLoggerIF logger)
                                                          throws java.io.IOException,
                                                                 PGMasterNotInServiceException

Handles input request packet from slave across a tunnel. Generates the response packet or throws an exception.

Parameters:

reqPacket - the request packet; never null

clientAddr - the tunnel client's address as seen by us; may be null if not applicable or available

cache - if not null, then selected routines with slow/expensive responses will not be serviced concurrently but concurrent calls will be vetoed quickly instead and expensive-to-compute values may be (partially) cached

logger - logging area for warnings, etc; never null.

Returns:

the response packet; never null

Throws:

java.io.IOException - in case of difficulty upstream

PGMasterNotInServiceException - if the upstream data source is down/unavailable

handleGetGenSecProps

private static ExhibitDataTunnelSource.RawPacket handleGetGenSecProps(SimpleExhibitPipelineIF source,
                                                                      ExhibitDataTunnelSource.RawPacket reqPacket)
                                                               throws java.io.IOException

Handle an incoming GetGenSecProps request; never null.

Throws:

java.io.IOException

handleSyncVariables

private static ExhibitDataTunnelSource.RawPacket handleSyncVariables(SimpleExhibitPipelineIF source,
                                                                     ExhibitDataTunnelSource.RawPacket reqPacket)
                                                              throws java.io.IOException

Handle an incoming SyncVariables request; never null.

Throws:

java.io.IOException

handleGetAllExhibitPropertiesDiff

private static ExhibitDataTunnelSource.RawPacket handleGetAllExhibitPropertiesDiff(SimpleExhibitPipelineIF source,
                                                                                   ExhibitDataTunnelSource.RawPacket reqPacket,
                                                                                   java.lang.String clientAddr,
                                                                                   ExhibitDataTunnelSource.HIRPCCache cache,
                                                                                   SimpleLoggerIF logger)
                                                                            throws java.io.IOException,
                                                                                   java.io.InterruptedIOException

Handle an incoming GetAllExhibitPropertiesDiff request; never null. Holds the 'slow response' lock while running to exclude other such intensive/slow responses concurrently.

Throws:

java.io.IOException

java.io.InterruptedIOException

handleGetAllExhibitProperties

private static ExhibitDataTunnelSource.RawPacket handleGetAllExhibitProperties(SimpleExhibitPipelineIF source,
                                                                               ExhibitDataTunnelSource.RawPacket reqPacket,
                                                                               java.lang.String clientAddr,
                                                                               ExhibitDataTunnelSource.HIRPCCache cache,
                                                                               SimpleLoggerIF logger)
                                                                        throws java.io.IOException

Handle an incoming GetAllExhibitProperties request; never null. Holds the 'slow response' lock while running to exclude other such intensive/slow responses concurrently.

Throws:

java.io.IOException

handleGetGenProps

private static ExhibitDataTunnelSource.RawPacket handleGetGenProps(SimpleExhibitPipelineIF source,
                                                                   ExhibitDataTunnelSource.RawPacket reqPacket)
                                                            throws java.io.IOException

Handle an incoming GetGenProps request; never null.

Throws:

java.io.IOException

handleGetStaticAttr

private static ExhibitDataTunnelSource.RawPacket handleGetStaticAttr(SimpleExhibitPipelineIF source,
                                                                     ExhibitDataTunnelSource.RawPacket reqPacket)
                                                              throws java.io.IOException

Handle an incoming GetStaticAttr request; never null.

Throws:

java.io.IOException

handleGetEventValues

private static ExhibitDataTunnelSource.RawPacket handleGetEventValues(SimpleExhibitPipelineIF source,
                                                                      ExhibitDataTunnelSource.RawPacket reqPacket)
                                                               throws java.io.IOException

Handle an incoming GetEventValues request; never null. Request is:

• A byte consisting of the ordinal of period/interval enum val.
• The UTF-8 representation of the name of the definition.
• The interval number (8 bytes).
• The serialised form of the request BitSet.

Result is:

• The in-order array of EventVariableValues (never null).

Throws:

java.io.IOException

handleGetVariables

private static ExhibitDataTunnelSource.RawPacket handleGetVariables(SimpleExhibitPipelineIF source,
                                                                    ExhibitDataTunnelSource.RawPacket reqPacket)
                                                             throws java.io.IOException

Handle an incoming GetVariables request; never null.

Throws:

java.io.IOException

handleSetVariables

private static ExhibitDataTunnelSource.RawPacket handleSetVariables(SimpleExhibitPipelineIF source,
                                                                    ExhibitDataTunnelSource.RawPacket reqPacket,
                                                                    java.lang.String clientAddr,
                                                                    SimpleLoggerIF logger)
                                                             throws java.io.IOException

Handle an incoming SetVariables request; never null. NOTE: for security/sanity reasons, reject any attempt to set where the value does not have a credible globalMap of exactly size == 1.

We'll validate the entire set of values, and reject the whole lot if any are dubious.

Throws:

java.io.IOException

handleGetVariable

private static ExhibitDataTunnelSource.RawPacket handleGetVariable(SimpleExhibitPipelineIF source,
                                                                   ExhibitDataTunnelSource.RawPacket reqPacket)
                                                            throws java.io.IOException

Handle an incoming GetVariable request; never null.

Throws:

java.io.IOException

handleGetRawFile

private static ExhibitDataTunnelSource.RawPacket handleGetRawFile(SimpleExhibitPipelineIF source,
                                                                  ExhibitDataTunnelSource.RawPacket reqPacket)
                                                           throws java.io.IOException

Handle an incoming GetRawFile request; never null. We satisfy the initial portion of over-length requests.

FIXME: decide whether to honour a false dontCache flag or not

Throws:

java.io.IOException

handleGetThumbnails

private static ExhibitDataTunnelSource.RawPacket handleGetThumbnails(SimpleExhibitPipelineIF source,
                                                                     ExhibitDataTunnelSource.RawPacket reqPacket)
                                                              throws java.io.IOException

Handle an incoming GetThumbnails request; never null.

Throws:

java.io.IOException

handleGetAllExhibitImmutableData

private static ExhibitDataTunnelSource.RawPacket handleGetAllExhibitImmutableData(SimpleExhibitPipelineIF source,
                                                                                  ExhibitDataTunnelSource.RawPacket reqPacket,
                                                                                  ExhibitDataTunnelSource.HIRPCCache cache,
                                                                                  SimpleLoggerIF logger)
                                                                           throws java.io.IOException

Handle an incoming GetAllExhibitImmutableData request; never null. Holds the 'slow response' lock while running to exclude other such intensive/slow responses concurrently.

Throws:

java.io.IOException

handleGetStratum

private static ExhibitDataTunnelSource.RawPacket handleGetStratum(SimpleExhibitPipelineIF source,
                                                                  ExhibitDataTunnelSource.RawPacket reqPacket)
                                                           throws java.io.IOException

Handle an incoming GetStratum request; never null.

Throws:

java.io.IOException