> ## Documentation Index
> Fetch the complete documentation index at: https://docs.metalntwx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitor and recover your integration

> Track chain and application progress, recover interrupted work, and verify database restoration

> [!NOTE]
>
> **What you'll be able to do**
>
> * Monitor chain progress against your application.
> * Recover interrupted operations using saved application records.
> * Run recovery drills for outages, lost responses, and duplicate delivery.

Your application updates its payment records by reading finalized blocks and
checking their events. Monitoring tracks whether Metal is finalizing new blocks
and whether your application is keeping up with them. Comparing their progress
helps you locate a delay in payment processing.

After an interruption, saved transaction hashes and verified events let your
application recover unfinished work. Its **cursor** stores the number and hash
of the last block it finished processing into its database, so it can resume
with the next block.

## Record the monitored configuration

Use the database from
[Verify and reconcile payments](https://docs.metalntwx.com/build/payments/verify-and-reconcile-payments.md). Record
the chain ID, endpoint label, token addresses, application contract addresses
and ABI versions, signing account address, and selected gas token. Use endpoint
labels in reports and keep credentials and signing keys in the application's
secret storage.

Resolve protocol addresses through
[Contract addresses](https://docs.metalntwx.com/reference/contract-addresses.md) and application addresses
through their deployment records. Verify code and configured permissions before
enabling writes. Probe the methods required by the flow, including
`eth_chainId`, `eth_getCode`, `eth_call`, `eth_getLogs`,
`eth_getTransactionReceipt`, and `consensus_status`. Record access restrictions,
log-range limits, history availability, and timeout behavior on the endpoint.
Resolve failed checks before relying on the affected operation.

For each workflow, identify the evidence that establishes its completion. A
direct payment needs a finalized transfer to the intended recipient. A payment
using a MIP20 hold also needs its capture result. Shielded payments, Clearing
payments, and bridged payments need their procedure's resolution, settlement,
or credit on the destination chain.

## Compare chain progress with application progress

Compare the latest executed block, the latest finalized block, and the last
block your application processed. The gap between execution and finality shows
how many blocks await finalization. **Processing lag** is the number of finalized
blocks your application still needs to process. If Metal has finalized block
100 and your application has processed through block 95, the processing lag is
five blocks.

The `sample` function measures these gaps in blocks. Pass the full endpoint URL
and the saved cursor's block number for the configured chain and token. Use the
cursor committed with the application's database updates to measure completed
processing.

```typescript theme={null}
async function rpc(rpcUrl: string, method: string, params: unknown[] = []) {
  const response = await fetch(rpcUrl, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ jsonrpc: "2.0", id: 1, method, params }),
    signal: AbortSignal.timeout(10_000)
  })
  const body = await response.json()
  if (!response.ok || body.error || body.result === undefined) {
    throw new Error(`RPC method failed: ${method}`)
  }
  return body.result
}

export async function sample(rpcUrl: string, processedHeight: bigint) {
  const started = performance.now()
  if (BigInt(await rpc(rpcUrl, "eth_chainId")) !== 63825n) throw new Error("Wrong network")
  const status = await rpc(rpcUrl, "consensus_status")
  const head = BigInt(status.head.number)
  const finalized = BigInt(status.finalized.number)
  if (head < finalized || processedHeight > finalized || processedHeight < 0n) {
    throw new Error("Inconsistent chain or processing cursor")
  }
  return {
    sampledAt: new Date().toISOString(),
    responseMs: performance.now() - started,
    executionHeight: head.toString(),
    finalizedHeight: finalized.toString(),
    finalityLagBlocks: (head - finalized).toString(),
    processedHeight: processedHeight.toString(),
    processingLagBlocks: (finalized - processedHeight).toString(),
    finalizedTimestampMs: status.finalized.timestamp_ms
  }
}
```

Save several samples at an application-chosen interval, such as five seconds for
a test. Compare execution and finalized heights over several samples and alert
when progress falls behind your application's threshold. Save that threshold and
the affected endpoint with the observation.

| Observation | Investigation |
| - | - |
| Request fails or exceeds the timeout | Endpoint access, credentials, RPC credit, network connectivity, and service health |
| Execution height stays unchanged | Compare a later sample and another configured endpoint before declaring chain progress stopped |
| Execution advances but finality does not | Keep business effects pending and escalate the finality delay |
| Finality advances but processing cursor does not | Processor errors, storage availability, ambiguous instructions, and event decoding |
| Cursor is beyond finality or its saved hash differs | Stop that stream and investigate network configuration, restored data, or provider inconsistency |
| Old signed attempt has no resolved outcome | Reconcile its hash and the signing account's transaction nonce before authorizing another attempt |

Verify the block hash stored with each cursor against the canonical block at
that height. A reset testnet can reuse a chain ID, so retain the network's
genesis block hash with the deployment configuration and compare it on
reconnect. Reconcile a reset explicitly before replaying records from an older
chain.

## Recover by the saved operation state

Load the saved instruction and transaction attempt from the database. Compare
its chain ID, signing account, destination address, call data, token, amount,
and transaction nonce with the intended operation. Read the receipt and
canonical block from the same endpoint.

| Saved state | Recovery action |
| - | - |
| Preparing, with no signed envelope | Establish that the former worker has stopped, inspect the signing account's transaction nonce and saved attempts, then reserve the signing account before resuming |
| Signed or submitted, with no known receipt | Query the saved hash, inspect pending and latest nonces, and rebroadcast identical bytes when appropriate |
| Included but not finalized | Keep the operation pending and repeat finality reads |
| Reverted transaction in a finalized block | Record the failure and fee, inspect the error, and require a new application decision before another attempt |
| Finalized success with wrong event | Stop fulfillment and investigate the actual execution |
| Finalized and matching, application update absent | Save the event and application update in one database transaction |
| Application update already present | Return the saved result |

For an RPC timeout, missing transaction or `nonce too low` response, follow the
[uncertain-submission procedure](https://docs.metalntwx.com/build/payments/verify-and-reconcile-payments.md#recover-an-uncertain-submission)
to resolve the original transaction before deciding whether another is needed.

Classify admission and execution failures using
[Error codes](https://docs.metalntwx.com/reference/error-codes.md) and
[Transactions](https://docs.metalntwx.com/reference/network-fundamentals/transactions.md). A deterministic policy rejection
requires correcting the instruction, policy or evidence before trying again.
Repeated execution failures can charge fees for each attempt.

## Run controlled failure drills

Use dedicated test accounts and a processor database on persistent storage. Keep
a known finalized payment and its receipt to test repeated processing. Simulate
network failures with controlled RPC responses in the test process, and record
which results came from those fixtures.

| Drill | How to trigger it | Required result |
| - | - | - |
| Lost submission response | In a test wrapper, send one authorized transaction and discard its response | Recover the saved hash and bytes without signing another transaction |
| Stale RPC | Return an earlier height from a read fixture | Detect lack of progress across samples and retain the application cursor |
| Delayed finality | Hold finalized height below a known receipt in a fixture | Keep the payment pending with no downstream effect |
| Signer unavailable | Make the signing adapter fail before producing a signature | Preserve the operation and require signer recovery before submission |
| Policy rejection | Simulate the configured rejected call or use its retained reverted receipt | Record the reason and leave the business effect absent |
| Duplicate event | Deliver the same verified event twice | One event row, one application effect, and one completed order |
| Crash during commit | Throw after event insertion and before order/cursor updates | Roll back every write, then apply once after restart |
| Processor outage | Stop discovery while the finalized head advances | Resume from the committed cursor and process every missed block |
| Database restoration | Restore a consistent backup and replay later finalized blocks | Recover the same completed orders without duplicate effects |

Pair the failure tests with the original finalized transaction and the
verification evidence retained by the processor.

## Restore a consistent database backup

For the SQLite store, create a consistent snapshot outside an active write
transaction. `VACUUM INTO` includes committed data from the database's journal
and requires a destination file that does not already exist:

```typescript theme={null}
db.exec("VACUUM INTO 'payments-backup.sqlite'")
```

Protect the backup like the live database because it contains signed
transactions and payment records. Keep its creation time, chain and genesis
hash, cursor heights and block hashes, schema version, and dependency lockfile.
The snapshot includes committed writes from the WAL that may be absent from the
active main database file.

Stop writers before restoration. Open the snapshot as a new database in a
separate directory, with no old WAL or shared-memory files. Verify the schema,
chain ID and genesis hash, saved cursor hashes, and uniqueness of recorded
events and application updates. Resume each stream from its stored cursor plus
one and replay through the finalized head. Compare completed order IDs and event
keys with the known live result.

An external service may have processed requests sent after the backup. Resend
pending delivery records with their original idempotency keys so the service
returns its existing result. If the backup predates a signed outgoing attempt,
its envelope may be lost while its transaction still exists. Reconcile
transaction nonces and transaction history for each signing account before
allowing new signing. Keep a separate persistent copy of signed attempts when
recovery must cover this case.

## Produce the operational report

Record the configuration, observations and recovery results:

* Chain ID and genesis hash, endpoint label, contract and ABI versions, and signing account roles.
* Capability probes with command, time, result, and access or history limits.
* Progress samples, alert thresholds, cursor hashes, and the last verified payment.
* Each drill's trigger, expected result, observed database state, and retained transaction hash.
* Backup location and creation time, restored cursor, replay range, and
  comparison of application records.
* Missing services, unavailable proof artifacts, untested live paths, and the
  next action required to resolve each one.

Repeat the affected checks when contracts, signing authority, endpoints, storage
or completion rules change.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.