S3 File Loader

When a Simulation object is used in the cloud, where there is no shared disk, KySim simulation files are fetched from S3 on demand instead of being read straight from a local folder. This is handled transparently by S3FileLoader; most users of kyoslib_py never need to interact with it directly.

Configuration

Simulation switches S3 fetching on only when both of these are true — mirroring the MATLAB Simulation constructor exactly:

1. The KySim settings file has <KySimInfo><FetchFromS3>1</FetchFromS3></KySimInfo>.
2. The KYOS_S3_BUCKET environment variable is set (the bucket name itself is never part of the settings file).

If the KySim settings file doesn't ask for S3 at all, Simulation behaves exactly as it always has: every file is read straight from the folder given in the KySim settings file, with no network calls at all.

If the settings file does ask for S3 (<FetchFromS3>1</FetchFromS3>) but KYOS_S3_BUCKET isn't set, that's a misconfiguration, not "off" — what happens next depends on whether this looks like a deployed container (see KYOS_S3_LOCAL_ROOT below for how that's detected):

• Deployed: raises immediately. A deployed container has no shared disk to fall back to, so continuing would only fail later, unrecognizably, deep inside whatever file is read first — this is exactly the failure mode that prompted this behaviour: a missing-bucket misconfiguration surfacing several files downstream as an unrelated model-specific error (e.g. an IndexError from an empty DataFrame), with no indication the real problem was kyoslib_py never actually reading anything.
• Not deployed: logs a warning and falls back to the settings file's literal path, exactly as if S3 had never been requested. There's no local-cache fallback to consider here — that cache can only exist if KYOS_S3_BUCKET was already set at some point to build it, so the literal path already covers the only case that can actually happen: a genuine on-prem server run.

Variable	Purpose	Default
KYOS_S3_BUCKET	Name of the S3 bucket to fetch simulation files from.	(none)
KYOS_S3_LOCAL_ROOT	Parent folder for the local cache.	see below

There is no region setting: boto3.client('s3') resolves the region itself, the same way every other AWS call on the same pod already does (AWS_REGION/AWS_DEFAULT_REGION, the AWS config profile, then instance metadata).

When KYOS_S3_LOCAL_ROOT is not set, S3FileLoader picks it automatically based on whether KUBERNETES_SERVICE_HOST is set (this is S3FileLoader's own default, so a standalone loader gets the same answer a Simulation-built one does). Kubernetes sets this itself on every pod in every cluster, so its presence means the job is genuinely running in the cloud rather than on the old tenant-host setup — the same question the MATLAB Simulation constructor's isdeployed() check answers, just via a different signal.

• If KUBERNETES_SERVICE_HOST is set, the cache is built one level above the current directory, so it never gets mixed in with the job's own input/output files.
• Otherwise (running locally, or still on the old tenant-host setup), the current directory is used directly.

An explicit KYOS_S3_LOCAL_ROOT always overrides both.

S3FileLoader can also be used on its own (outside of Simulation), in which case it falls back to reading a FETCH_FROM_S3 environment variable (true/1 to enable) instead of an XML flag — see the class docstring below.

list_dir() names the files and subfolders inside a directory, whether they live locally or only in S3. Not called from anywhere in kyoslib_py itself yet, but models built on top of Simulation use it — for example, to figure out a commodity's simulated price granularity (hourly, half-hourly, or quarter-hourly) from which filenames actually show up in its spot simulation folder, rather than assuming a fixed naming pattern up front.

Required IAM policy

Once S3 fetching is enabled, the pod or container running the model needs an IAM role with:

• s3:GetObject on arn:aws:s3:::<bucket>/simulation/* — to download simulation files.
• s3:ListBucket on arn:aws:s3:::<bucket>, scoped to the simulation/* prefix. This is required for correct day-to-day behaviour, not just the one-time startup sanity check. Without it, a request for a file that is genuinely missing from S3 comes back 403 Forbidden instead of 404 Not Found — and the loader cannot tell that apart from a real permissions problem, so it raises an error instead of treating it as "no data for this file". That fallback is exactly what lets things like load_sims_for_all_months stop cleanly at the last available month, so a GetObject-only role will break in ways that only show up once a file is actually missing. list_dir() also needs it whenever it falls back to S3.

Do not confuse s3:ListBucket with s3:ListBuckets (plural). s3:ListBucket only lists the contents of one named bucket — the sanity check above never does more than that, so granting it is safe. s3:ListBuckets is a different, account-wide permission that lists every bucket the credentials can see — kyoslib_py never calls it, it should not be granted, and it is not required for anything in this repo.

kyoslib_py never configures or grants these permissions itself — it only ever uses whichever credentials boto3's own default credential chain finds (environment variables locally, the pod's IAM role in the cloud, or an AWS SSO profile for local development — see Connecting to AWS S3).

Known limitations

• The startup check can reject a bucket that is only temporarily empty. The one-time sanity check above raises immediately if nothing at all exists yet under the configured prefix — this is right for catching a genuinely wrong bucket/prefix, but it also means a brand new client's very first job, before anything has been uploaded for them yet, would fail at Simulation construction rather than later. There is no MATLAB equivalent to this check (MATLAB never verifies the bucket up front — it only finds out per file, when it tries to load one).
• kyoslib_py does not hardcode a region. Unlike MATLAB, which always passes a region to its S3 client (defaulting to "eu-central-1" unless overridden), kyoslib_py leaves this to boto3.client('s3'), which resolves it the same way MATLAB's AWS SDK for Java does: AWS_REGION, then AWS_DEFAULT_REGION, then an AWS config profile, then EC2/EKS instance metadata. Every pod that needs a non-default region already sets one of these for its other AWS calls, so nothing extra is needed here.

Documentation

S3FileLoader

Transparently download KySim simulation files from S3 to a local cache folder, so that code written for a shared on-prem disk keeps working unchanged when it runs in a container with no shared disk.

The loader does three things:

1. rebase() rewrites a shared-disk folder path (e.g. from the KySim XML) onto a local cache folder. This only needs to run once, when a Simulation object is built.
2. ensure_local_file() checks whether one exact file is already on disk and, if not, downloads it from S3. This has to run every time, right before a file is opened.
3. list_dir() names the files and subfolders inside a directory, whether they exist locally or only in S3.

When fetch_from_s3 is False (the default), all three are pass-throughs and nothing in kyoslib_py behaves any differently than it does today.

fetch_from_s3=True with no bucket configured is a misconfiguration, not "off" - what happens next depends on whether this looks like a deployed container:

• Deployed (see _is_deployed()): raises immediately. There is no shared disk to fall back to, so continuing would only fail later, unrecognizably, deep inside whatever file is read first.
• Not deployed: logs a warning and turns fetch_from_s3 off, falling back to the literal on-prem path from the settings file - the same behaviour as if S3 had never been requested at all. There is no local-cache fallback to consider here: the local cache can only exist if KYOS_S3_BUCKET was already set at some point to build it.

Configuration is read from environment variables so it works the same way whether a Simulation object is built from an XML file or directly from a dictionary:

• FETCH_FROM_S3 ("true"/"1" to enable, anything else or unset disables it)
• KYOS_S3_BUCKET (the S3 bucket name, e.g. "kyos-simulations" - "s3://kyos-simulations" also works, since that's the form MATLAB expects)
• KYOS_S3_LOCAL_ROOT (optional; where to build the local cache folder - see local_root_parent below for the default when this is unset)

The AWS region is never configured here - boto3.client('s3') resolves it itself (AWS_REGION/AWS_DEFAULT_REGION, the AWS config profile, then instance metadata), the same as every other AWS pod already relies on for every other AWS call it makes.

IAM policy required by the pod/container running the model, once fetch_from_s3 is enabled:

• s3:GetObject on arn:aws:s3:::<bucket>/<path_anchor>/* (every file download)
• s3:ListBucket on arn:aws:s3:::<bucket>, scoped to the <path_anchor>/* prefix. This is required for correct behaviour, not just the one-time startup check below: without it, a request for a missing file gets back "403 Forbidden" instead of "404 Not Found", which ensure_local_file() cannot tell apart from a real permissions problem - so a genuinely missing file raises instead of being treated as "no data for this file", exactly the fallback this whole feature depends on (e.g. load_sims_for_all_months in Simulation stopping cleanly at the last available month).

Attributes:

Name	Type	Description
fetch_from_s3	bool	Whether files should be fetched from S3.
bucket	str	Name of the S3 bucket to fetch simulation files from.
path_anchor_local	str	The path segment (e.g. "simulation") that marks where the "interesting" part of a local folder path begins.
path_anchor_s3	str	The equivalent path segment on the S3 side. Usually the same as path_anchor_local; pass an empty string to store objects at the bucket root with no anchor prefix.
local_root_parent	Path	The folder under which the local cache is built.
local_root	Path	local_root_parent / path_anchor_local. Every rebased folder path lives under here.

Examples:

>>> from kyoslib_py.s3_file_loader import S3FileLoader
>>> loader = S3FileLoader(fetch_from_s3=True, bucket='kyos-simulations')
>>> spot_folder = loader.rebase('/kyos/data/simulation/Spot/KyPlant')
>>> spot_file = loader.ensure_local_file(spot_folder + '/DailySpot_TTF.mat')

__init__(fetch_from_s3=None, bucket=None, path_anchor='simulation', path_anchor_s3=None, path_anchor_local=None, local_root_parent=None, verbose=False)

Constructor of the S3FileLoader class.

Parameters:

Name	Type	Description	Default
fetch_from_s3	bool	Whether to fetch missing files from S3. Defaults to the FETCH_FROM_S3 environment variable.	None
bucket	str	S3 bucket name. Defaults to the KYOS_S3_BUCKET environment variable. Required when fetch_from_s3 is True. An "s3://" prefix (the form MATLAB expects) is accepted and stripped automatically.	None
path_anchor	str	Shared path segment used for both the local cache and the S3 key layout, unless overridden below. Defaults to "simulation".	'simulation'
path_anchor_s3	str	Path segment used on the S3 side, if different from path_anchor. Pass an empty string for "no anchor prefix on S3".	None
path_anchor_local	str	Path segment used on the local side, if different from path_anchor.	None
local_root_parent	str	Parent folder for the local cache. Defaults to the KYOS_S3_LOCAL_ROOT environment variable; if that's unset too, one level above the current directory when KUBERNETES_SERVICE_HOST is set (a real Kubernetes cluster sets this on every pod automatically), otherwise the current directory unchanged.	None
verbose	bool	Print a line for every file fetched from S3.	False

Raises:

Type	Description
S3FileLoaderError	If fetch_from_s3 is True but no bucket is configured and this is a deployed container (no shared disk to fall back to), if no local path anchor is configured, or if the bucket/prefix cannot be reached at all.

ensure_local_dir(local_dir)

Ensure every file in a directory is available locally.

Downloads the whole matching S3 prefix when the directory does not exist locally, or exists but contains no files. Ports MATLAB's EnsureLocalDir (used there to fetch a job's whole ForwardCurves directory in one go). Not called from anywhere in kyoslib_py yet, because the directory-based forward-curve reader it would serve has not been ported to Python yet either -- get_fwd_curve here reads one named CSV file at a time instead. This is not speculative: add the caller when that reader is ported, not before.

Parameters:

Name	Type	Description	Default
local_dir	str or Path	Path to the local directory, already rebased (i.e. a descendant of local_root).	required

Raises:

Type	Description
S3FileLoaderError	If nothing exists under the matching S3 prefix either, or if a download fails for any other reason (wrong credentials, no network, ...).

ensure_local_file(local_path)

Ensure a file exists locally, downloading it from S3 first if it is missing.

Parameters:

Name	Type	Description	Default
local_path	str or Path	Path to the local file, already rebased (i.e. a descendant of local_root).	required

Returns:

Type

Description

Path

local_path, as a Path. When fetch_from_s3 is False this is returned completely unchanged (no filesystem check, no network call). When fetch_from_s3 is True, the file is downloaded first if it was missing; if the object genuinely does not exist in S3 either, the path is returned as-is, still missing, so that the caller's existing "file not found" handling (the same one used for a missing file on-prem) takes over exactly as it does today.

Raises:

Type	Description
S3FileLoaderError	If the download fails for any reason other than the object not existing (wrong credentials, no network, wrong bucket, ...). These are configuration problems, not "no data for this file", so they are always raised rather than treated like a missing file.

list_dir(local_dir)

List the names of the files and subfolders directly inside a directory, falling back to S3 when nothing has been downloaded into that directory yet.

Downstream models (e.g. KyBattery, KyPlant) call this to work out which simulation files actually exist for a commodity, then infer their time granularity (hourly, half-hourly, or quarter-hourly) from which filenames show up - rather than assuming a fixed naming pattern up front.

Parameters:

Name	Type	Description	Default
local_dir	str or Path	Path to the local directory, already rebased (i.e. a descendant of local_root).	required

Returns:

Type	Description
list[str]	Names of the entries directly inside local_dir, sorted alphabetically. Empty if the directory does not exist, locally or in S3.

Raises:

Type	Description
S3FileLoaderError	If the S3 listing fails (wrong credentials, no network, ...).

rebase(folder_path)

Rewrite a shared-disk folder path onto the local cache folder.

Call this once per folder, when a Simulation object is built. It has no effect (returns its input unchanged) when fetch_from_s3 is False.

Parameters:

Name	Type	Description	Default
folder_path	str	A folder path that contains path_anchor_local as one of its path segments, e.g. the KySim XML's SpotSimFolder.	required

Returns:

Type	Description
str	folder_path unchanged, if fetch_from_s3 is False or folder_path is empty. Otherwise the same folder, rebased under local_root.

Raises:

Type	Description
S3FileLoaderError	If fetch_from_s3 is True and path_anchor_local is not one of the path segments in folder_path.

S3FileLoaderError

Bases: RuntimeError

Raised when a KySim simulation file cannot be fetched from S3 for a reason other than the file simply not existing (for example: wrong credentials, wrong bucket, or no network).

Deliberately not a subclass of OSError. A missing file on-prem raises OSError, and some calling code catches OSError broadly to mean "no data for this commodity/month, move on". An expired SSO token or a wrong bucket is a different problem - the job is misconfigured, not missing data - so it must not be catchable by the same broad except OSError.