org.hd.d.pg2k.webSvr.util
Class WebUtils

java.lang.Object
  org.hd.d.pg2k.webSvr.util.WebUtils

public final class WebUtils
extends java.lang.Object

Web-server-related utility functions. This is for algorithms only of interest to Web apps, often JSPs.

One advantage of having code here rather than in-line in a JSP is that is is pre-compiled off-line for speed and robustness; code here is also easier to test.

VOTER_LOCATION_STATS_EVENT_DEF

public static final SimpleVariableDefinition VOTER_LOCATION_STATS_EVENT_DEF

Name of event/variable to which we post a voter's approximate geo location.

VOTER_LOCATION_STATS_EVENT_PREFIX

public static final java.lang.String VOTER_LOCATION_STATS_EVENT_PREFIX

Prefix of event value for voter's approximate geo location.

See Also:

Constant Field Values

_gPE_vars

private static final SimpleVariableDefinition[] _gPE_vars

System variables tried, in order, for a "popular" exhibit; private to getPopularExhibit(). We put the download var first to get a decent rate of update since this value changes quite frequently.

We don't put the vote var first so as to reduce the temptation to "throw" a vote to get an exhibit shown on the front page.

CHECK_FOR_SPIDERS_BY_UA

private static final boolean CHECK_FOR_SPIDERS_BY_UA

If true then check for bots/spiders by UA (User-Agent); note that clients can forge their UA.

See Also:

Constant Field Values

spiderUAName1stWordsLC

private static final java.util.Set<java.lang.String> spiderUAName1stWordsLC

Immutable Set of known spider/bot UA strings; should probably be moved to a text/properties file. This is a set of lower-cased first (space/tab/bracket-delimited) words from the UA strings.

The names consist only of non-regex-metacharacters in the set [a-z'_-], so are safe to embed in a regex.

UA_TERMINATOR_CHARS

private static final java.lang.String UA_TERMINATOR_CHARS

Set of characters taken as a main-part terminator in a User-Agent header, including whitespace. This should be usable as the separator arg to StringTokenizer, and in a regex when wrapped in "[]" square brackets.

See Also:

Constant Field Values

UA_BOT_PATTERNS

private static final java.lang.String UA_BOT_PATTERNS

Extra case-insensitive patterns matched in UA names, "|"-separated, or null if none. Essentially, anything alphanumeric-ish ending in "bot", or anything alphanumeric-ish containing "spider".

See Also:

Constant Field Values

UA_REGEX

public static final java.util.regex.Pattern UA_REGEX

Case-insensitive regex match for all non-empty UA names from spiderUAName1stWordsLC; null if not checking UA names. Made public to enable some unit testing.

_rPFS_CACHE_PNAME

private static final java.lang.String _rPFS_CACHE_PNAME

Name of Boolean attribute in request we cache result of requestProbablyFromSpider() by.

See Also:

Constant Field Values

_isBot_cache

private static final MemoryTools.CacheMiniMap<java.lang.String,java.lang.Boolean> _isBot_cache

LRU cache from (common) whole UAs to "bot"ness to save some repeated/slow String parsing; never null when checking for bots by UA. We're prepared to discard all of this under memory stress as we only have to work this out at most once per request for example.

TODO: consider some form of incremental auto-expiry even when not full since keys can be relatively large (eg something like SimpleLRUMap + AutoExpirable)

_lastNotLightlyLoaded

private static volatile long _lastNotLightlyLoaded

Private flag for isLightlyLoaded() to note time when we were last non lightly loaded. Once we discover that we are not lightly loaded we set this to the current time, and then do not re-test for a little while in case the testing itself adds to the load.

Initially zero.

Marked volatile for thread-safe lock-free access.

_lastResetLA

private static volatile long _lastResetLA

Time that we last reset the target load average to zero to restart our load ramp-up. Initially 'now' in encourage a gentle start.

Marked volatile for thread-safe lock-free access.

LOAD_AVERAGE_RAMP_UP_TIME

private static final int LOAD_AVERAGE_RAMP_UP_TIME

Time over which to ramp up load limit in isLightlyLoaded(), ms; strictly positive. Should be at least the 60s time claimed in the documentation for OperatingSystemMXBean.getSystemLoadAverage() to dampen CPU-load flapping, especially when emerging from power-conservation mode.

TN_AVAIL_CACHE

public static final boolean TN_AVAIL_CACHE

If true then exhibitHasThumbnail() will try to cache its results. This should avoid us having to load thumbnails into memory just to see if they exist or not.

See Also:

Constant Field Values

TN_AVAIL_CACHE_SIZE_MIN_TARGET

private static final int TN_AVAIL_CACHE_SIZE_MIN_TARGET

The target thumbnail availability cache minimum size, strictly positive. Should be large enough that most thumbnail availability requests will be caught by it regardless of exhibit set size, but small enough to limit memory requirements to something reasonable. Should be *much* larger than the full set of thumbnails ever seen on one page and/or that might be in our "popular" page set so as to avoid thrashing fruitlessly.

We are prepared to give up all cache entries rather than cause an OOME.

See Also:

Constant Field Values

tnHTMLCacheKey

private static final DataSourceBean.AEPLinkedKey tnHTMLCacheKey

Private key used by exhibitHasThumbnail(); never null. Made AEP linked to more be able to accurately size it for the current AEP, and ensure that it is periodically rebuilt/refreshed from scratch.

_gACSI_cache

private static final LRUMapAutoSizeForHitRate<java.lang.String,java.lang.Object> _gACSI_cache

Private cache for getAndCacheStaticImage(); never null. Is a map from root-relative URL to a BufferedImage or SoftReference to a BufferedImage.

Thread-safe LRU-managed limited-size memory-sensitive map.

We are prepared to discard everything if very short of memory.

All getAndCacheStaticImage() activity is synchronized on this instance.

VOTE_PRO_PARAM_NAME

public static final java.lang.String VOTE_PRO_PARAM_NAME

Name of additional parameter to record if this is a vote "for" ("pro"). Value of the parameter is to be "true" or "false".

See Also:

Constant Field Values

VOTE_CON_PARAM_NAME

public static final java.lang.String VOTE_CON_PARAM_NAME

Name of additional parameter to record if this is a vote "against" ("con"). Value of the parameter is to be "true" or "false".

See Also:

Constant Field Values

metadataCacheKey

private static final DataSourceBean.AEPLinkedKey metadataCacheKey

Private key used by getCatPageExhibitMetaDataHTML(); never null.

sDictMD

public static final Compact7BitString.StaticDictionary sDictMD

Static dictionary used by getCatPageExhibitMetaDataHTML() for compression of in-memory data; never null. The static dictionary content should be reviewed after any major format changes, though this is not a correctness issue, only a matter of compression efficiency.

_inlineableInXHTML

private static final MemoryTools.CacheMiniMap<Tuple.Pair<ExhibitMIME.ExhibitTypeParameters,java.lang.String>,java.lang.Boolean> _inlineableInXHTML

LRU thread-safe private cache mapping from exhibit type and Accept header to acceptability of that MIME type for inlining in XHTML mobile text. We assume that the Accept headers will be more or less constant for a given device, and probably constant between instances of the device, so we have enough entries to cover the likely different types of device using the Gallery at any one time.

We take care to avoid using huge Accept values in keys to avoid DoS-style issues.

We're prepared to discard this entirely when short of memory.

TODO: consider some form of incremental auto-expiry even when not full since keys can be relatively large (eg something like SimpleLRUMap + AutoExpirable)

_getCategoryTreeFilterBeanKey

private static final DataSourceBean.AEPLinkedKey _getCategoryTreeFilterBeanKey

Private key used by getCategoryTreeFilterBean(); never null.

MOBILE_REGEX_FLAGS

private static final int MOBILE_REGEX_FLAGS

Flags for User-Agent pattern matching checking for mobile phones.

See Also:

Constant Field Values

MOBILE_REGEX_DMB_1_20110812

private static final java.util.regex.Pattern MOBILE_REGEX_DMB_1_20110812

Mobile-browser detection regex 1 c/o detectmobilebrowser.com 2010/08/12.

MOBILE_REGEX_DMB_2_20110812

private static final java.util.regex.Pattern MOBILE_REGEX_DMB_2_20110812

Mobile-browser detection regex 2 c/o detectmobilebrowser.com 2010/08/12.

ALLOW_SNEAKY_HTTP_CONCURRENCY

public static final boolean ALLOW_SNEAKY_HTTP_CONCURRENCY

If true then allow some "sneaky" browser concurrency. For browsers that do not always pipeline by default but do allow fetches from different-named hosts in parallel (eg the biggies such as IE6/IE7, FF1/1.5/2 as of 2006Q4), then we can possibly help throughput by fetching some images (etc) from a different name or the literal IP address for this same host. (The literal IP address has the benefit of needing no DNS lookup.)

The trick is that where the user has arrived at the site with a name other than the local mirror name (or literal IP), and the URI that we were going to use was a rrURL (root-relative URL), then we can prepend the local mirror name/IP to make a new absolute URL that the browser may be prepared to fetch from concurrently.

This has to be done completely consistently for any given item, eg a static "page-furniture" image, to avoid defeating cacheing.

See Also:

Constant Field Values

LITERAL_IP_SNEAKY_HTTP_CONCURRENCY

private static final boolean LITERAL_IP_SNEAKY_HTTP_CONCURRENCY

Iff true, do sneaky concurrency with a literal IP address. This is instead of using the local mirror name, and avoids any extra DNS lookup by the client, and can be used even when the client is visiting a mirror explicitly.

See Also:

Constant Field Values

SNEAKY_HTTP_CONCURRENCY_REQ_ATTR_CACHE

private static final java.lang.String SNEAKY_HTTP_CONCURRENCY_REQ_ATTR_CACHE

Request attribute to cache getOptionalSneakyConcurrencyRRURLPrefix() response. Useful if sneaky concurrency is attempted several times in one response. If a non-null value is stored against the attribute, it's used.

See Also:

Constant Field Values

TEST_PNAME_PREFIX

private static final java.lang.String TEST_PNAME_PREFIX

Prefix of all user testimonials in the global properties.

See Also:

Constant Field Values

_gUT_cache

private static final java.util.Map<java.lang.String,java.util.List<java.lang.String>> _gUT_cache

Logically immutable cached lists of testimonal Strings by language (not locale); never null. Private to getUserTestimonial().

No null keys, not null/empty values.

Small fixed size.

Thread-safe (and supporting reasonable concurrency if possible).

trailDataCacheKey

private static final DataSourceBean.AEPLinkedKey trailDataCacheKey

Private key used by getTrialData(); never null.

_awc_CacheKey

private static final DataSourceBean.AEPLinkedKey _awc_CacheKey

Private key used by approxWordCount(); never null.

_iPCE_vars

private static final java.util.List<SimpleVariableDefinition> _iPCE_vars

Events to be examined by isPopularCatalogueEntry(); never null. These must all have VLONG data stored.

findLatestCodeBundleKey

private static final DataSourceBean.AEPLinkedKey findLatestCodeBundleKey

Private key used by findLatestCodeBundle(); never null.

CODE_SECTION_DIR

private static final java.lang.String CODE_SECTION_DIR

Name of the section/dir in which code/doc bundles are filed.

See Also:

Constant Field Values

WEB_TRAFFIC_CYCLE_MS

private static final int WEB_TRAFFIC_CYCLE_MS

Target traffic cycle time for recentTrafficLowForTypicalCycle() in milliseconds; strictly positive. This is typically a week or multiple thereof to allow for the usual major cycle in traffic flows, and to detect weekends and holiday dips for example.

See Also:

Constant Field Values

WEB_TRAFFIC_SAMPLE_PERIOD

private static final EventPeriod WEB_TRAFFIC_SAMPLE_PERIOD

Sample interval used by WEB_TRAFFIC_CYCLE_MS; not null. A smaller interval gives a finer-grained response at the cost of more work. Typically MEDIUM or LONG would be used to have a response time around the hour mark.

WEB_TRAFFIC_MIN_SAMPLES

private static final int WEB_TRAFFIC_MIN_SAMPLES

Minimum number of samples to trust when deciding traffic levels; strictly positive. A value between about 4 and whatever a whole day is (inclusive) probably makes sense.

See Also:

Constant Field Values

WEB_TRAFFIC_SAMPLE_PERIODS

private static final int WEB_TRAFFIC_SAMPLE_PERIODS

Integer number of LONG sample periods used to cover WEB_TRAFFIC_CYCLE_MS; strictly positive.

WEB_TRAFFIC_TRIM_LEADING_ZEROS

private static final boolean WEB_TRAFFIC_TRIM_LEADING_ZEROS

If true then ignore any leading run of (oldest) zero traffic counts to give initial faster response for new server.

See Also:

Constant Field Values

rTLFLC_resultKey

private static final DataSourceBean.UnlinkedKey rTLFLC_resultKey

Non-AEP-linked cache key for recentTrafficLowForTypicalCycle() result; non-null. Stored value against key is (immutable) paid of interval number and boolean 'traffic low' flag. This usually is usually from WEB_TRAFFIC_SAMPLE_PERIOD key, but a more cautious shorter-term key with disjoint values can be used instead to avoid repeated recomputation in the face of possible-transient problems such as with start-up and I/O.

WebUtils

private WebUtils()

Prevent construction of an instance.

getPopularExhibit

public static Name.ExhibitFull getPopularExhibit(DataSourceBean dsb,
                                                 ExhibitMIME.ExhibitTypeParameters type,
                                                 java.util.Collection<java.lang.String> excludeFullNames,
                                                 boolean beQuick)

Gets name of "popular" exhibit, possibly filtered by type; null if none available. Tries to pick a "popular" exhibit by looking at one recently voted for, or downloaded, etc, in the system variables, and that has both thumbnails available where they are possible.

This rejects exhibits with a below-par (negative) rating.

Using the system variables should mean that this can pick up values set from any mirror, etc, fairly quickly.

This cannot guarantee to return a non-null value, but any value that it does return is a current, valid exhibit.

Parameters:

dsb - handle on the system variables and data; never null

type - if not null only exhibits of this type are candidates

excludeFullNames - if non-null, any exhibits included by full name are not candidates to be returned

beQuick - if true then don't spend too long trying to calculate this but instead give up quickly if need be (so as not to block page generation for example)

requestProbablyFromSpider

public static boolean requestProbablyFromSpider(javax.servlet.http.HttpServletRequest request)

Attempt to determine quickly if the current request is probably from a spider/bot (ie not a human). NB: This does not attempt to distinguish between good spiders (ie bona fide SEs) and bad/broken/rude bots/scrapers/spiders.

This tries to distinguish human from non-human, at least in part to conserve (prime interactive) resources for humans, and does not claim to be perfect.

This returns true if there is no (valid) referring page (and this visitor is not new to the site, ie has recently pulled another page), though some browsers/firewalls may routinely block this info, and some referrals, eg to target="_blank", may show no Referer value.

This should be quick as most or all requests may be tested with this.

TODO: Should expand this to test against well-known IP addresses.

TODO: Should include a behavioural element, eg mindless, rapid, pauseless browsing.

Parameters:

request - the incoming request; never null

Returns:

true if the requesting client is probably a bot, false if probably a human

requestProbablyReferredFromExternalSite

public static java.lang.String requestProbablyReferredFromExternalSite(javax.servlet.http.HttpServletRequest request)

Attempt to detect if the current request has been referred from an external Web site. This checks if the "Referer" is apparently from gallery mirrors or aliases.

Note that since this info can be forged, or disabled/knobbled for security reasons, this is not completely reliable.

A missing "Referer" will be taken as indicating an "external" reference, and will catch most well-behaved spiders as a result.

Since we may have to do some string parsing this may not be very fast, but it should not be very slow either.

Even if there is more than one "Referer" header we will only look at one.

Parameters:

request - the incoming request; never null

Returns:

null if referred from a Gallery site/host/alias, "" if unparsable or no referring URL, else normalised (lowercase, stripped of common prefixes, etc) referring host's name

makeExhibitURL

public static java.net.URL makeExhibitURL(java.lang.CharSequence exhibitName,
                                          javax.servlet.http.HttpServletRequest request,
                                          DataSourceBean vars)
                                   throws java.net.MalformedURLException

Generate full URL for exhibit given exhibit name; never null. This may take account of such factors as which servers are up, how heavily loaded, etc, in order to give best throughput and perform automatic load balancing, to give the best possible user experience.

In order to do this well we may need the request details, so they should be supplied if possible. These should be the client's request to a Gallery site, else null.

Defaults to generic main host if specific better mirror (etc) cannot be identified.

Throws:

java.net.MalformedURLException

makeExhibitRRURL

public static java.lang.String makeExhibitRRURL(java.lang.CharSequence exhibitName)

Generate root-relative URL for exhibit given exhibit name; never null.

makeThumbnailURL

public static java.net.URL makeThumbnailURL(java.lang.CharSequence exhibitName,
                                            boolean std)
                                     throws java.net.MalformedURLException

Generate full URL for thumbnail/sample given exhibit name and standard/small selector; never null.

Throws:

java.net.MalformedURLException

makeThumbnailRRURL

public static java.lang.String makeThumbnailRRURL(java.lang.CharSequence exhibitName,
                                                  boolean std)

Generate root-relative URL for thumbnail/sample given exhibit name and standard/small selector; never null.

makeCatPageURL

public static java.net.URL makeCatPageURL(java.lang.CharSequence exhibitName,
                                          java.lang.String mediaTypeSuffix)
                                   throws java.net.MalformedURLException

Generate full URL for catalogue page given exhibit name; never null. This should always generate the canonical/"official" form of the URL, eg not including mirrors or aliases.

The media-type suffix (eg ".html" or ".wml") must be supplied.

Throws:

java.net.MalformedURLException

makeCatPageRRURL

public static java.lang.String makeCatPageRRURL(java.lang.CharSequence exhibitName,
                                                java.lang.String mediaTypeSuffix)

Generate root-relative URL for catalogue page given exhibit name; never null. This should take account of such factors as which servers are up, how heavily loaded, etc.

The media-type suffix (eg ".html" or ".wml") must be supplied.

isOverloaded

public static boolean isOverloaded(javax.servlet.ServletContext ctxt)

Returns true if this Web server is overloaded (eg for bandwidth or CPU). If this server is clearly overloaded then this routine returns true, and it is possible to start trimming UI features to reduce load, eg the number of search results shown.

If not overloaded or not possible to tell, this returns false.

(If the context is null then this routine may have to return false.)

This may gather information from a number of sources, but in the main the ThroughputFilter's data will be used. We may explicitly set the overload flag here if we detect the system to be overloaded to make it quicker for us and others on a subsequent call; this will get overwritten by TMF when it next gets to assess load.

If true then the UI and application should attempt to use less bandwidth and CPU time than normal, perhaps by using cheaper algorithms than usual (eg less comprehensive searches) or a less-graphics-intensive UI.

This may default to true while the server is starting up to ensure that as little extra CPU (for example) as possible is consumed while the server is compiling JSPs, loading classes, etc.

This is intended to be relatively cheap to call, since it may get called frequently, for example especially when the system is busy, and/or at start-up before JIT compilation (eg HotSpot) has kicked in.

isLightlyLoaded

public static boolean isLightlyLoaded(javax.servlet.ServletContext ctxt)

Returns true if the site seems to be lightly loaded (CPU, bandwidth, etc). In case of difficulties/confusion this reports false.

When running well, the system should over around the normal / light-load boundary.

This routine is quite careful and conservative, and will only return true if this server and the host system both seem to be lightly loaded and stable by all the appropriate metrics.

This never reports the system to be lightly loaded if it is low on power (eg on a nearly-expired battery) so as to avoid unnecessary background work.

Where this can check 'uptime' then it tries to enforce a modest CPU ramp-up over approximately the 60s-or-so of OperatingSystemMXBean.getSystemLoadAverage() to limit rapid fluctuations in CPU load (and, for example, power consumption).

This routine is designed to be called frequently, ie is reasonably fast and efficient.

exhibitsHaveThumbnail

public static java.util.BitSet exhibitsHaveThumbnail(DataSourceBean dataSource,
                                                     java.util.List<Name.ExhibitFull> exhibitNames,
                                                     boolean standard,
                                                     boolean forceCreation)

Test if the given exhibits have thumbnails (locally) available; never null. This is suitable for a bulk check, eg before rendering a table, and may be internally parallelised to overcome I/O latency, etc.

Parameters:

exhibitNames - non-null list of full exhibit names; not altered by this routine and must not be changed by the caller while this routine is running

Returns:

a bit in the result set for each thumbnail that definitely exists in the requested size; an unset bit may represent "not known"

exhibitHasThumbnail

public static boolean exhibitHasThumbnail(DataSourceBean dataSource,
                                          Name.ExhibitFull exhibitName,
                                          boolean standard,
                                          boolean forceCreation)

Test if the given exhibit has a thumbnail (locally) available. Does not try to force one to be made if one is not extant (or in cache) unless the forceCreation argument is true.

Since this is likely to be testing what is in (local) cache, its results can be considered at best a (good) hint.

We do not cache entirely negative answers (ie that an exhibit has no thumbnails) as this may change if we overcome (eg) a temporary resource issue. But we assume the converse, ie that once we have a thumbnail then it will always be available.

Always returns false for exhibit types that do not support thumbnails.

Parameters:

dataSource - the data source; never null

exhibitName - the full exhibit name; must be valid (eg non-null)

standard - if true, tests for the presence of a standard-size

forceCreation - if true, we can try to force creation/fetch of the underlying thumbnail if not already available locally

Returns:

true if exhibit definitely has/had one/both thumbnails, false if unknown or thumbnails are not currently available

makeHTMLInlineImageThumbnailURL

public static java.lang.String makeHTMLInlineImageThumbnailURL(DataSourceBean dataSource,
                                                               Name.ExhibitFull exhibitName,
                                                               boolean std,
                                                               boolean rrURLOnly,
                                                               java.awt.Dimension tnDim,
                                                               boolean dontCreateTn)
                                                        throws java.io.IOException

Get thumbnail image URL to embed in HTML page (usually JPEG/GIF/PNG); null if none available. By preference uses purpose-built thumbnail, else tries to use image itself if small enough.

Has to be passed a dataSource and a full exhibit name.

Returns null if no suitable thumbnail URL can be generated.

This may cache its results, in particular assuming that once a particular thumbnail becomes available that it does not go away again.

Parameters:

tnDim - (output argument) if not null and result is not null, is filled in with the thumbnail dimensions

dontCreateTn - if true, don't force creation of a thumbnail if one is not already readily available

rrURLOnly - if true then only generate a root-relative URL, else an absolute URL at a CDN (or with sneaky concurrency) is allowed and either form may be returned

Throws:

java.io.IOException

getExhibitVariantRange

public static java.util.List<Name.ExhibitFull> getExhibitVariantRange(java.util.List<Name.ExhibitFull> allExhibitNames,
                                                                      java.util.Comparator<Name.ExhibitFull> comparator,
                                                                      int thisExhibitIndex)

Find the set of exhibits with the same subject as the indicated one. Given a List of String exhibit names sorted by ExhibitPropsGlobalImmutable.SMART_ORDER (or possibly just by ExhibitPropsGlobalImmutable.SUBJECT_ORDER), the SUBJECT_ORDER comparator, and the index to a given item in that List, finds all the items surrounding that have the same SUBJECT_ORDER, ie that differ only in attribute words and are variants on the same exhibit. (The result will always contain the input item.)

The List must not contain nulls, all entries must be valid exhibit names, and the List must be sorted implicitly with the comparator passed in. The input index must be within bounds.

This does not alter the List passed in. The return value is an unmodifiable sublist of the input.

This returns the sublist of variants on the indicated exhibit; this will degenerate to just containing the input exhibit name if there are no other exhibit names for the same exhibit.

This assumes that the number of variants of any given exhibit is small, and so a linear search is used.

If the List does not support efficient bidirectional movement and seeks then this routine will be very inefficient.

minimalUniqueENTitlePrefix

public static java.lang.CharSequence minimalUniqueENTitlePrefix(java.util.List<Name.ExhibitFull> exhibits,
                                                                int ourIndex)

Computes a minimal human-readable immutable unique prefix of an exhibit short name to distinguish given exhibit from most others. Uniqueness is not guaranteed, but this is meant to give a reasonably short result that a human can read in the title of a page, for example.

If inputs are bad then this will try to fail gently with an empty-string result rather than an exception to make it robust if called directly from JSP code, for example.

This is passed a List of exhibits sorted in a "smart" order that sorts mainly on the file component of the name, probably in a case-insensitive way.

This routine finds a short word prefix that (case-insensitively) is different from neighbouring exhibits and is thus (depending on the overall sort order) probably the shortest unique prefix amongst the entire collection.

(If no unique value is possible, the entire prefix is returned.)

If the List of size 0 we return the empty string and do not use the index parameter at all. This simplifies use where the list may transiently be empty during start-up.

This routine may run very slowly if the List argument does not support efficient random seeking.

The result is designed to be used in headings and other display text such as image alt/title attributes.

TODO: fix inefficient double-parsing of main words...

Parameters:

exhibits - sorted (smart-ish sorted) list of exhibit names (String value); must not change while routine is running

ourIndex - index (strictly positive) of the exhibit whose abbreviated name we wish to produce; must be within the List

Returns:

"" in case of invalid arguments, else some whole-word prefix of the name

getAndCacheStaticImage

public static java.awt.image.BufferedImage getAndCacheStaticImage(boolean copyResult,
                                                                  java.lang.String resourceRRURL,
                                                                  boolean forceToARGBTrueColour,
                                                                  javax.servlet.ServletContext context)
                                                           throws java.lang.IllegalStateException

Get BufferedImage containing expanded image loaded as static resource from WAR; never null. Loaded on demand and cached statically, via a SoftReference to allow the system to reclaim memory rather than OOM.

There may be an adverse effect on system behaviour if many large images are cached in memory; this may be mitigated by storing them via a SoftReference so that the memory can be recycled automatically.

This method is internally synchronized to allow only one image load/decode to happen at once to conserve CPU and memory (and other) resources.

If the image is indexed and forceToARGBTrueColour is true then we expand it to true-colour to make it possible to add markings.

This uses our built-in mediahandler classes to decode the image, so the image type must be one that we have a decoder for.

This may not handle alpha correctly in all cases.

Under memory pressure this may release cached resources whether hard or soft cached.

Beware: since a reference is returned rather than a copy, be careful not to adjust the returned image unintentionally.

Parameters:

copyResult - if true, we force the result to be a copy of the cached value to avoid accidentally changing the cached copy; this may force a change in colour model and/or discarding properties

resourceRRURL - root-relative URL (eg "/my/image.gif") of source image in WAR; must not be null and must be a type (and with a file extension) that we have a MIME mediahandler for

forceToARGBTrueColour - if true, and the source image is not ARGB true-colour, then we force conversion to ARGB true-colour before cacheing to make it easier to draw on the image

context - gives context for WAR from which to load the raw base clickable-map image

Returns:

image, possibly a copy, possibly with a converted colour model

Throws:

java.lang.IllegalStateException - if the image is not loadable

generateUserDataPointID

public static java.lang.String generateUserDataPointID(java.lang.String prefix,
                                                       javax.servlet.http.HttpServletRequest request)

Generate a unique key for the given HTTP request; returns null if not possible to generate. For example, we use this to help limit the number of times that a given user is asked to vote.

Generates a string starting with the given prefix and some leading portion of the client's IP address...

It is not fatal if this conflates users, but it is more useful that this never thinks one user is more than one to avoid pestering them too much (or letting a spider inject too much noise for example).

Parameters:

prefix - unique prefix to the generated key; non-empty, non-null

request - the user's request

createAndFileVoteListener

public static java.lang.String createAndFileVoteListener(Name.ExhibitFull exhibitFullName,
                                                         javax.servlet.http.HttpServletRequest request,
                                                         SimpleVariablePipelineIF vars)

Create and post the handler for a vote if possible and returns the unique listenerID. This replaces any existing listener for this voter.

This returns null if we could not create a listener.

If the handler is invoked, then this registers a dummy (inactive) handler to delay the next request to the user to vote.

Parameters:

exhibitFullName - valid exhibit name; never null

request - client's HTTP request; never null

vars - where stats updates are posted; never null

Returns:

listenerID, or null if one could not be generated

getCategoryListSortedAsHTML

public static final java.lang.String getCategoryListSortedAsHTML(DataSourceBean dsb,
                                                                 javax.servlet.http.HttpServletRequest request,
                                                                 LocaleBeanBase localeBean,
                                                                 boolean asList)
                                                          throws java.io.IOException

Get sorted, hyperlinked HTML i18n text list of exhibit categories; never null. The list is sorted:

1. By dictionary-order i18n localised title.

We may flag categories as good or bad if significantly so.

Parameters:

asList - if true, entries are preceded by <li> else they are followed by <br />\r\n;

Throws:

java.io.IOException

getCategoryListSortedAsHTML

public static final java.lang.String getCategoryListSortedAsHTML(DataSourceBean dsb,
                                                                 javax.servlet.http.HttpServletRequest request,
                                                                 LocaleBeanBase localeBean,
                                                                 java.lang.String entrySeparator)
                                                          throws java.io.IOException

Get sorted, hyperlinked HTML i18n text list of exhibit categories; never null. The list is sorted:

1. By dictionary-order i18n localised title.

We may flag categories as good or bad if significantly so.

Parameters:

entrySeparator - text (followed by CRLF) to terminate entries; null if entries are to be wrapped with li tags

Throws:

java.io.IOException

isDisconnectedSlave

public static final boolean isDisconnectedSlave(DataSourceBean dsb)

Returns true if this seems to be a slave disconnected from the master. This instance may, for example, not wish to collect votes from users if the votes may get discarded without getting to the master.

isPrecacheRequest

public static boolean isPrecacheRequest(javax.servlet.http.HttpServletRequest request)

Returns true if this request is apparently a precacheing request, eg from a "Web accelerator". This is true if a client (such as FireFox) is "reading ahead" but it may be the case that no real human gets to see the content.

See https://developer.mozilla.org/en/Link_prefetching_FAQ

getCatPageExhibitMetaDataHTML

public static java.lang.Object getCatPageExhibitMetaDataHTML(DataSourceBean dsb,
                                                             Name.ExhibitFull exhibitName)

Generates HTML form of exhibit metadata, "" if no such metadata for the specified exhibit; never null. The result is keyed to both the DataSourceBean and the exhibitName.

Cached values are discarded when the AEP changes.

The computed HTML is locale-invariant, which is why cacheing is viable.

The toString() method should be called on the result to get the String HTML text, which may be a String or some other internal representation.

Parameters:

dsb - valid non-null DataSourceBean

exhibitName - valid non-null exhibit full name

getCatPageExhibitMetaDataHTMLRaw

public static java.lang.String getCatPageExhibitMetaDataHTMLRaw(Name.ExhibitFull exhibitName,
                                                                AllExhibitProperties aep)

Computes the raw data for getCatPageExhibitMetaDataHTML() without cacheing; never null.

Returns:

"" if there is no metadata for this exhibit

getCatPageExhibitMetaDataHTMLFuture

public static java.util.concurrent.Future<?> getCatPageExhibitMetaDataHTMLFuture(DataSourceBean dsb,
                                                                                 Name.ExhibitFull exhibitName)

As for getCatPageExhibitMetaDataHTML(), but will compute a missing value asynchronously; never null. If the value is already computed then it is available immediately, else this attempts to spin off task compute the value, and get() will block until the value is ready/computed.

If the target thread pool is full the computation will be done synchronously, ie in this thread blocking this call until complete.

The toString() method should be called on the result to get the String text.

hyphenateHTMLText

public static final java.lang.String hyphenateHTMLText(java.lang.String s)

Hyphenate long HTML text (that contains zero or more `-' characters and little or no whitespace). Replaces hyphens ('-') with spaces to allow a browser to wrap the text.

Usually used with exhibit names or fragments of such names.

canInlineInXHTMLPage

public static boolean canInlineInXHTMLPage(ExhibitMIME.ExhibitTypeParameters exhibitType,
                                           javax.servlet.http.HttpServletRequest request)

Returns true if the given MIME-type can always be inlined in an XHTML (mobile) page. If the type argument is null, this returns false.

This always allows JPEG and GIF, but may also allow other (image) types listed in an incoming "Accept" header.

canInlineInHTMLPage

public static boolean canInlineInHTMLPage(ExhibitMIME.ExhibitTypeParameters exhibitType)

Returns true if the given MIME-type can always be inlined in an HTML page. If the argument is null, this returns false.

getCategoryTreeFilterBean

public static TreeFilterBean getCategoryTreeFilterBean(DataSourceBean dsb,
                                                       java.lang.CharSequence category)

Get selected by-category TreeFilterBean from entire exhibit set; never null. Used for the "by category" exhibit tree view and elsewhere.

This data is cached linked to the DSB (which in passing ensures that it can be dropped automatically under extreme memory stress).

The category name is primarily checked for syntactic validity, not for actual presence in the AEP.

isBrowserOnMobileDevice

public static boolean isBrowserOnMobileDevice(javax.servlet.http.HttpServletRequest request)

Return true if client appears likely to be a mobile device (browser sniffing). This attempts to detect a small-display, CPU- and bandwidth- constrained device, that might benefit from being sent small XHTML pages rather than standard HTML.

This examines the User-Agent and is unlikely to be completely reliable.

Thanks for the regex to http://detectmobilebrowser.com/

getOptionalSneakyConcurrencyRRURLPrefix

public static java.lang.String getOptionalSneakyConcurrencyRRURLPrefix(javax.servlet.http.HttpServletRequest request)

Get optional prefix for rrURL for extra "sneaky" browser concurrency. This can only apply if:

• ALLOW_SNEAKY_HTTP_CONCURRENCY is true.
• This host has a mirror prefix, ie so that there is an "alternate" name/IP for this host.
• The incoming request hostname is not this local mirror hostname.
• We do recognise the form (a vhost for) the URL in the request.

else this routine always returns "".

This basically only works if the user is using a "generic" URL, but that is more likely to be a machine far away (ie with large RTT), so extra concurrency to try to overcome latency is especially helpful.

This slightly inflates the HTML that the client will see but only if using a generic URL.

May inflate the number of concurrent connections back to this host, but usually only by 1 or 2 at most.

Note that this scheme does not rely on any other host being up, nor having exactly the same content as us.

This is not a technique for distributing load.

Returns:

"" or the http://mirror-... name (with no trailing slash) for this host

getUserTestimonial

public static java.lang.String getUserTestimonial(LocaleBeanBase l)

Get short user quote/testimonial at random from those available for the given locale; never null but may be "". This finds a quote, if any, suitable for the current locale (infact, currently just the language is matched), picked randomly from those available, or "" if none is available.

The text is HTML/XML safe, and is pure 7-bit printable ASCII, with any non-ASCII characters encoded as HTML/XML entities.

Parameters:

l - the required locale; never null

getTrailData

public static TrailData getTrailData(DataSourceBean dsb,
                                     Name.ExhibitFull trailExhibitFullName)

Get the TrailData for a given trail exhibit (by full name); null if none. This caches the result in the DSB, linked to the AEP, so never retaining data for expired trails.

(This may negatively cache failure to load TrailData (for a while) for efficiency.)

Safe to apply to an arbitrary/unvetted exhibit name, even an invalid/null value.

approxWordCount

public static final int approxWordCount(DataSourceBean dsb,
                                        Name.ExhibitFull fullExhibitName)

Compute (crude) estimate of words in catalogue page for given exhibit; non-negative. This is designed to be reasonably fast, though not necessarily amazingly accurate, and is intended to help decide how many ad blocks a page may reasonably support.

This may cache its results against the AEP instance.

This counts 'non-furniture' words, ie those originating from the data itself, including the exhibit name, exhibit description, tree AKA/description, etc, with different constituents possibly weighted differently.

For simplicity, this does its computations based on the default site language, even if there may be significant variation in apparent word count for other localisations.

Returns:

zero in case of difficulty (eg exhibit does not exist), else approximate (positive) word count

isPopularCatalogueEntry

public static boolean isPopularCatalogueEntry(SimpleVariablePipelineIF vars,
                                              java.lang.CharSequence exhibitFullName)

Returns true iff the named exhibit and/or catalogue page is popular (well visited/downloaded). Uses the history to decide if a catalogue page and its exhibit are frequently visited/downloaded (wrt other catalogue pages globally and locally).

Parameters:

vars - source of event history; never null

exhibitFullName - full exhibit name; never null

Returns:

true if popular, false otherwise

findLatestCodeBundle

public static Name.ExhibitFull findLatestCodeBundle(DataSourceBean dsb,
                                                    java.lang.String prefix)

Returns full exhibit name for latest version of a code bundle, or null if none. This locates the latest (with a major-minor-micro versioning) bundle in the 'code' section, for the given prefix, or null of none.

The author and extension are ignored for selection purposes.

For example, for the prefix/argument 'javadoc', if the code section includes the files 'javadoc-1-2-3-DHD.zip' and javadoc-1-10-1-ANON.zip' this will return 'code/javadoc-1-10-1-ANON.zip'.

To be found a bundled archive name must be exactly of the form:
prefix-major-minor-micro-AUTH.XTN
where the major, minor and micro components are (small, non-negative) integers. There must be no attributes present.

(Note that a '-' is appended to the supplied prefix.)

This may cache the results against the AEP instance, since the lookup may happen may times and we may have to search through a fair amount of data for each lookup. Note: this does not cache negative results in part to bound the amount of space that can be consumed.

Parameters:

dsb - current data source; never null

prefix - legitimate short-name as bundle name; never null nor empty

abortIfNotModifiedSince

public static boolean abortIfNotModifiedSince(long lastModified,
                                              javax.servlet.http.HttpServletRequest request,
                                              javax.servlet.http.HttpServletResponse response)

Returns true (and sets SC_NOT_MODIFIED status) iff the caller should avoid sending a GET response body. Intended to be be called by a servlet handling a GET/HEAD operation before most headers are set or any response body is sent/commited.

Parameters:

lastModified - last time this entity changed, or -1 if not known / not applicable

request - never null (unless lastModified == -1)

response - never null (unless lastModified == -1)

Returns:

true if SC_NOT_MODIFIED has been set and servlet should return immediately without sending a body, false if no status set and body may still have to be sent.

abortIfETagMatchOrNotModifiedSince

public static boolean abortIfETagMatchOrNotModifiedSince(java.lang.String eTag,
                                                         long lastModified,
                                                         javax.servlet.http.HttpServletRequest request,
                                                         javax.servlet.http.HttpServletResponse response)

Returns true (and sets SC_NOT_MODIFIED status) iff the caller should avoid sending a GET response body. Intended to be be called by a servlet handling a GET/HEAD operation before most headers are set or any response body is sent/commited.

Parameters:

eTag - valid single ETag token, strong or weak, for the page; null if not known / not applicable

lastModified - last time this entity changed; -1 if not known / not applicable

request - never null (unless eTag == null and lastModified == -1)

response - never null (unless eTag == null and lastModified == -1)

Returns:

true if SC_NOT_MODIFIED has been set and servlet should return immediately without sending a body, false if no status set and body may still have to be sent.

computeCacheMaxAgeMSFromTimestamp

public static long computeCacheMaxAgeMSFromTimestamp(long timestamp,
                                                     javax.servlet.ServletContext ctxt,
                                                     GenProps gp)

Compute a suitable cache expiry time for a usually slowly-changing object (ms); non-negative. Treats the item as if almost static in terms of rate of change (but constrains the result to be no longer than the minimum for static objects).

Makes the cache time usually a significant multiple of the interval between rechecks of exhibit immutable data as this is expected to change relatively slowly.

Extend to a reasonable fraction of the underlying item's time since last change capped to the maximum allowed for static content, essentially replicating a common heuristic from browsers.

Increase it if the system is conserving/busy so as to reduce future server load.

getNewsflashHTML

public static java.lang.String getNewsflashHTML(GenProps gp)

Get "newsflash" HTML for the main site front page, or "" if none; never null. This is retrieved from the GenProps.

recentTrafficLowForLatestCycle

public static boolean recentTrafficLowForLatestCycle(DataSourceBean dsb)

Returns true if the last full period or so had low traffic compared to the last larger or so. Generally true if traffic by some metric, local or global, is in the bottom quartile, typically for a recent hour or thereabouts (to allow a nimble response) compared to a weekly cycle. Can be used to stabilise ad revenue for example by switching in extra ads when traffic is low, for example at weekends.

The calculation result is cached (against the dsb) to reduce CPU effort.

Parameters:

dsb - data source bean from which to retrieve stats; never null

Returns:

false unless recent traffic was clearly below normal over a typical cycle