Monitor and recover your integration

What you’ll be able to do

Monitor chain progress against your application.
Recover interrupted operations using saved application records.
Run recovery drills for outages, lost responses, and duplicate delivery.

Your application updates its payment records by reading finalized blocks and checking their events. Monitoring tracks whether Metal is finalizing new blocks and whether your application is keeping up with them. Comparing their progress helps you locate a delay in payment processing. After an interruption, saved transaction hashes and verified events let your application recover unfinished work. Its cursor stores the number and hash of the last block it finished processing into its database, so it can resume with the next block.

Record the monitored configuration

Use the database from Verify and reconcile payments. Record the chain ID, endpoint label, token addresses, application contract addresses and ABI versions, signing account address, and selected gas token. Use endpoint labels in reports and keep credentials and signing keys in the application’s secret storage. Resolve protocol addresses through Contract addresses and application addresses through their deployment records. Verify code and configured permissions before enabling writes. Probe the methods required by the flow, including eth_chainId, eth_getCode, eth_call, eth_getLogs, eth_getTransactionReceipt, and consensus_status. Record access restrictions, log-range limits, history availability, and timeout behavior on the endpoint. Resolve failed checks before relying on the affected operation. For each workflow, identify the evidence that establishes its completion. A direct payment needs a finalized transfer to the intended recipient. A payment using a MIP20 hold also needs its capture result. Shielded payments, Clearing payments, and bridged payments need their procedure’s resolution, settlement, or credit on the destination chain.

Compare chain progress with application progress

Compare the latest executed block, the latest finalized block, and the last block your application processed. The gap between execution and finality shows how many blocks await finalization. Processing lag is the number of finalized blocks your application still needs to process. If Metal has finalized block 100 and your application has processed through block 95, the processing lag is five blocks. The sample function measures these gaps in blocks. Pass the full endpoint URL and the saved cursor’s block number for the configured chain and token. Use the cursor committed with the application’s database updates to measure completed processing.

async function rpc(rpcUrl: string, method: string, params: unknown[] = []) {
  const response = await fetch(rpcUrl, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ jsonrpc: "2.0", id: 1, method, params }),
    signal: AbortSignal.timeout(10_000)
  })
  const body = await response.json()
  if (!response.ok || body.error || body.result === undefined) {
    throw new Error(`RPC method failed: ${method}`)
  }
  return body.result
}

export async function sample(rpcUrl: string, processedHeight: bigint) {
  const started = performance.now()
  if (BigInt(await rpc(rpcUrl, "eth_chainId")) !== 63825n) throw new Error("Wrong network")
  const status = await rpc(rpcUrl, "consensus_status")
  const head = BigInt(status.head.number)
  const finalized = BigInt(status.finalized.number)
  if (head < finalized || processedHeight > finalized || processedHeight < 0n) {
    throw new Error("Inconsistent chain or processing cursor")
  }
  return {
    sampledAt: new Date().toISOString(),
    responseMs: performance.now() - started,
    executionHeight: head.toString(),
    finalizedHeight: finalized.toString(),
    finalityLagBlocks: (head - finalized).toString(),
    processedHeight: processedHeight.toString(),
    processingLagBlocks: (finalized - processedHeight).toString(),
    finalizedTimestampMs: status.finalized.timestamp_ms
  }
}

Save several samples at an application-chosen interval, such as five seconds for a test. Compare execution and finalized heights over several samples and alert when progress falls behind your application’s threshold. Save that threshold and the affected endpoint with the observation.

Observation	Investigation
Request fails or exceeds the timeout	Endpoint access, credentials, RPC credit, network connectivity, and service health
Execution height stays unchanged	Compare a later sample and another configured endpoint before declaring chain progress stopped
Execution advances but finality does not	Keep business effects pending and escalate the finality delay
Finality advances but processing cursor does not	Processor errors, storage availability, ambiguous instructions, and event decoding
Cursor is beyond finality or its saved hash differs	Stop that stream and investigate network configuration, restored data, or provider inconsistency
Old signed attempt has no resolved outcome	Reconcile its hash and the signing account’s transaction nonce before authorizing another attempt

Verify the block hash stored with each cursor against the canonical block at that height. A reset testnet can reuse a chain ID, so retain the network’s genesis block hash with the deployment configuration and compare it on reconnect. Reconcile a reset explicitly before replaying records from an older chain.

Recover by the saved operation state

Load the saved instruction and transaction attempt from the database. Compare its chain ID, signing account, destination address, call data, token, amount, and transaction nonce with the intended operation. Read the receipt and canonical block from the same endpoint.

Saved state	Recovery action
Preparing, with no signed envelope	Establish that the former worker has stopped, inspect the signing account’s transaction nonce and saved attempts, then reserve the signing account before resuming
Signed or submitted, with no known receipt	Query the saved hash, inspect pending and latest nonces, and rebroadcast identical bytes when appropriate
Included but not finalized	Keep the operation pending and repeat finality reads
Reverted transaction in a finalized block	Record the failure and fee, inspect the error, and require a new application decision before another attempt
Finalized success with wrong event	Stop fulfillment and investigate the actual execution
Finalized and matching, application update absent	Save the event and application update in one database transaction
Application update already present	Return the saved result

For an RPC timeout, missing transaction or nonce too low response, follow the uncertain-submission procedure to resolve the original transaction before deciding whether another is needed. Classify admission and execution failures using Error codes and Transactions. A deterministic policy rejection requires correcting the instruction, policy or evidence before trying again. Repeated execution failures can charge fees for each attempt.

Run controlled failure drills

Use dedicated test accounts and a processor database on persistent storage. Keep a known finalized payment and its receipt to test repeated processing. Simulate network failures with controlled RPC responses in the test process, and record which results came from those fixtures.

Drill	How to trigger it	Required result
Lost submission response	In a test wrapper, send one authorized transaction and discard its response	Recover the saved hash and bytes without signing another transaction
Stale RPC	Return an earlier height from a read fixture	Detect lack of progress across samples and retain the application cursor
Delayed finality	Hold finalized height below a known receipt in a fixture	Keep the payment pending with no downstream effect
Signer unavailable	Make the signing adapter fail before producing a signature	Preserve the operation and require signer recovery before submission
Policy rejection	Simulate the configured rejected call or use its retained reverted receipt	Record the reason and leave the business effect absent
Duplicate event	Deliver the same verified event twice	One event row, one application effect, and one completed order
Crash during commit	Throw after event insertion and before order/cursor updates	Roll back every write, then apply once after restart
Processor outage	Stop discovery while the finalized head advances	Resume from the committed cursor and process every missed block
Database restoration	Restore a consistent backup and replay later finalized blocks	Recover the same completed orders without duplicate effects

Pair the failure tests with the original finalized transaction and the verification evidence retained by the processor.

Restore a consistent database backup

For the SQLite store, create a consistent snapshot outside an active write transaction. VACUUM INTO includes committed data from the database’s journal and requires a destination file that does not already exist:

db.exec("VACUUM INTO 'payments-backup.sqlite'")

Protect the backup like the live database because it contains signed transactions and payment records. Keep its creation time, chain and genesis hash, cursor heights and block hashes, schema version, and dependency lockfile. The snapshot includes committed writes from the WAL that may be absent from the active main database file. Stop writers before restoration. Open the snapshot as a new database in a separate directory, with no old WAL or shared-memory files. Verify the schema, chain ID and genesis hash, saved cursor hashes, and uniqueness of recorded events and application updates. Resume each stream from its stored cursor plus one and replay through the finalized head. Compare completed order IDs and event keys with the known live result. An external service may have processed requests sent after the backup. Resend pending delivery records with their original idempotency keys so the service returns its existing result. If the backup predates a signed outgoing attempt, its envelope may be lost while its transaction still exists. Reconcile transaction nonces and transaction history for each signing account before allowing new signing. Keep a separate persistent copy of signed attempts when recovery must cover this case.

Produce the operational report

Record the configuration, observations and recovery results:

Chain ID and genesis hash, endpoint label, contract and ABI versions, and signing account roles.
Capability probes with command, time, result, and access or history limits.
Progress samples, alert thresholds, cursor hashes, and the last verified payment.
Each drill’s trigger, expected result, observed database state, and retained transaction hash.
Backup location and creation time, restored cursor, replay range, and comparison of application records.
Missing services, unavailable proof artifacts, untested live paths, and the next action required to resolve each one.

Repeat the affected checks when contracts, signing authority, endpoints, storage or completion rules change.

Prepare an integration

Process payments

Control spending

Deploy and automate contracts

Use data and signatures

Connect other chains

Operate your integration

Monitor and recover your integration

Record the monitored configuration

Compare chain progress with application progress

Recover by the saved operation state

Run controlled failure drills

Restore a consistent database backup

Produce the operational report

​Record the monitored configuration

​Compare chain progress with application progress

​Recover by the saved operation state

​Run controlled failure drills

​Restore a consistent database backup

​Produce the operational report

Record the monitored configuration

Compare chain progress with application progress

Recover by the saved operation state

Run controlled failure drills

Restore a consistent database backup

Produce the operational report