Operationalisation (Part 2)

Part 2 runs the cycle on a schedule, publishes valid forecasts, quarantines invalid ones, and alerts on trouble. An EventBridge rule triggers a Step Functions state machine every six hours, and the state machine runs one SageMaker Processing job on the ECR container.

flowchart TD
  EB[EventBridge rule wm3-6h] --> SF[Step Functions wm3-cycle]
  SF -->|createProcessingJob.sync| P[Processing job: ml.g5.xlarge, ECR image]
  P --> G[fetch latest available GFS f000]
  G --> I[preprocess and infer]
  I --> V{core output valid?}
  V -->|yes| F[(forecasts/)]
  F --> L[latest.json]
  V -->|no| Q[(quarantine/)]
  V -->|no| S1[SNS invalid-output message]
  P --> CW[CloudWatch metrics]
  SF -->|job failure after retries| S2[SNS failure message]

How it runs

• The container pulls weights from S3 at start, so nothing is baked into the image.
• Step Functions retries the job twice, then routes a failure to SNS.
• A valid cycle writes under forecasts/ and replaces latest.json; an invalid one writes under quarantine/ and leaves the pointer untouched.

Two independent signals

• cycle_success=1 means the job finished as designed.
• output_valid=0 is a separate quality signal, so a bad forecast is quarantined and alerted rather than promoted.
• Two alarms watch these conditions independently.

Failure behavior

Condition	Storage result	Metrics	Notification
NaN, Inf, or failed physical checks	Artifacts under quarantine/; pointer unchanged	output_valid=0, cycle_success=1	Direct SNS plus invalid-output alarm
GFS, inference, writer, or upload exception	No pointer promotion	cycle_success=0	Step Functions retries twice, then SNS
No cycle_success for six hours	No storage change	Missing datapoint treated as breaching	Failure and staleness alarm
Experimental precipitation	A valid atmospheric forecast still publishes	precip_reliable=false, tail diagnostics	Recorded in CloudWatch

Metrics and dashboard

• The WeatherMesh3 namespace records per-stage timings, cycle_success, output_valid, nonfinite_count, temperature and pressure ranges, peak winds, and the humidity, dewpoint, and precipitation diagnostics.
• The WeatherMesh3 dashboard shows these together.
• SNS topic wm3-alerts has a confirmed email subscription.