Repository navigation
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical flaws allow Hub to modify decision-runtime state, permit filesystem writes outside the private cache, and let expired workers update jobs.
15 open issues
Attach connector identities to runtime hosts, not Hub · New Gate permanent deletion on decision-runtime enablement · New Isolate decision-runtime tables from Hub's database role · New Check lease expiry using database time after locking · New Validate and sanitize candidate paths before materialization · New Bound archive expansion during taskset registration and resolution · New Allow established runtime usage kinds · New Return the original create result for identical retries · New Use the provider-neutral hard-budget error code · New Validate malformed status and cancel started executions · New Attempt remote cancellation when recovery fails · New Enforce inference idempotency and detect key conflicts · New Preserve measured usage when the budget is exceeded · New Test the real connector input and context propagation · New Make the stale-listing test simulate a missing blob version · New
What changed in this PR
Adds a customer-operated Azure reference bundle for the decision runtime, building on the provider-neutral lifecycle, budget, and generation contracts from #297 and #298.
Changes:
- Adds Azure storage, PostgreSQL state, training, inference, identity, and telemetry connectors with mocked contract coverage.
- Passes inference generations and cooperative cancellation signals to workers.
- Adds optional Azure infrastructure and operating guidance; live cloud verification remains outstanding.
| File | Description |
|---|---|
packages/learning/environments/brand-check/tests/test_runtime.py |
Adds worker cancellation and timeout coverage. |
packages/learning/environments/brand-check/tests/test_azure_connectors.py |
Adds mocked Azure connector and conformance coverage. |
packages/learning/environments/brand-check/runtime.py |
Propagates inference generations and worker cancellation. |
packages/learning/environments/brand-check/README.md |
Documents Azure adapters and operational boundaries. |
packages/learning/environments/brand-check/azure_connectors.py |
Implements the Azure connector bundle. |
packages/learning/environments/brand-check/__init__.py |
Exposes Azure APIs through lazy exports. |
infrastructure/README.md |
Documents provisioning and required cloud verification. |
infrastructure/azure/variables.tf |
Adds decision-runtime configuration inputs. |
infrastructure/azure/tests/deployment.tftest.hcl |
Checks decision storage, access grants, and identity wiring. |
infrastructure/azure/storage.tf |
Adds decision storage, retention, and permanent-delete support. |
infrastructure/azure/roles.tf |
Defines decision identities and scoped access grants. |
infrastructure/azure/outputs.tf |
Exports storage, job, and identity coordinates. |
infrastructure/azure/application.tf |
Adds decision identities and configuration to Hub. |
Review effort: Max (adaptive)
Comment on lines
+80
to
+82
| var.enable_decision_runtime ? [ | ||
| azurerm_user_assigned_identity.decision_runtime[0].id, | ||
| azurerm_user_assigned_identity.decision_inference[0].id, |
| delete_retention_policy { | ||
| days = 14 | ||
| days = 14 | ||
| permanent_delete_enabled = true |
|
|
||
|
|
||
| AZURE_POSTGRES_SCHEMA = """ | ||
| CREATE TABLE IF NOT EXISTS replica_decision_jobs ( |
| SET record = %s::jsonb, version = version + 1 | ||
| WHERE scope_key = %s AND job_id = %s AND version = %s | ||
| AND lease_token = %s AND lease_owner_kind = %s | ||
| AND lease_owner_id = %s AND lease_expires_at > %s |
Comment on lines
+1493
to
+1495
| directory = self._cache_root / candidate_id | ||
| directory.mkdir(parents=True, exist_ok=True) | ||
| path = directory / name |
Comment on lines
+1655
to
+1669
| try: | ||
| self.cancellation_recovery_hook(execution_id, context) | ||
| except Exception as error: | ||
| raise ConnectorError( | ||
| "azure_training_cancellation_unrecorded", | ||
| "Azure training cancellation could not be scheduled for recovery.", | ||
| category="internal", | ||
| ) from error | ||
| for attempt in range(3): | ||
| try: | ||
| self.transport.cancel(execution_id, bearer_token=token) | ||
| return | ||
| except Exception: | ||
| if attempt < 2: | ||
| self.sleep(0.1 * (attempt + 1)) |
Comment on lines
+1765
to
+1769
| dict( | ||
| request, | ||
| endpoint=self.endpoint_name, | ||
| expectedGeneration=expected_generation, | ||
| ), |
Comment on lines
+2104
to
+2119
| if ( | ||
| context.budget_mode == "hard" | ||
| and context.max_credits is not None | ||
| and sum( | ||
| item.quantity | ||
| for item in usage | ||
| if item.kind == "training_credits" and item.unit == "credits" | ||
| ) | ||
| > context.max_credits | ||
| ): | ||
| raise ConnectorError( | ||
| "credit_budget_exceeded", | ||
| "Azure reported usage above the hard credit budget.", | ||
| category="integrity", | ||
| ) | ||
| return training |
Comment on lines
+824
to
+830
| answers = self.connector.predict( | ||
| adapter, | ||
| dict(request, expectedGeneration=1), | ||
| context=context, | ||
| credential=credential, | ||
| ) | ||
| return dict(answers, operationId=context.operation_id) |
Comment on lines
+971
to
+978
| for name in tuple(blobs.blobs): | ||
| blobs.upload_blob( | ||
| name, | ||
| blobs.blobs[name]["data"], | ||
| overwrite=True, | ||
| metadata=blobs.blobs[name]["metadata"], | ||
| ) | ||
| blobs.versions[name].insert(0, {"id": "already-deleted"}) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Why
Replica needs a practical Azure reference for the customer-operated decision runtime. The runtime must keep private training data, model artifacts, credentials, and infrastructure charges inside the customer's Azure subscription while preserving the provider-neutral lifecycle merged in #297 and #298.
What changes
This adds Azure adapters for private immutable Blob artifacts, durable PostgreSQL job and promotion state, bounded GPU job execution, generation-protected inference, managed identity credentials, and redacted operational signals.
Terraform adds the private decision container, split workload identities, an exact non-delete Blob data role, broker and inference grants, retention policy, and deployment coordinates. A separate withdrawal identity permanently deletes selected Blob versions; runtime and training identities cannot delete them.
The connector and infrastructure tests use mocks and require no cloud credentials. A real Azure account is still required to verify regional GPU quota, the customer-provisioned job and broker, RBAC propagation, private connectivity, measured cost, cold starts, permanent withdrawal deletion, restore, outage, quota-exhaustion, and teardown behavior. This pull request does not claim a live Azure deployment pass.