Diagnostics
Something is wrong.
22 symptoms and 58 causes, indexed by what you are actually seeing rather than by which part of the protocol it belongs to. Nearly all of these fail silently, which is why they persist.
Almost nothing here raises an error. An unindexed subgraph returns an empty array, a stale allocation stops paying, a stopped payment agent keeps serving queries, an overpriced cost model looks exactly like an absence of demand.
If you are waiting for something to go red before investigating, these are the failures that will wait with you.
Reading data
My query returns an empty array
- 1
The subgraph has not indexed that far yet
Query indexing status alongside the data and compare the indexed block against chain head.
- 2
The deployment failed at some block and stopped
Check deployment health. A failed subgraph keeps answering correctly for everything up to where it stopped, and raises nothing.
- 3
One indexer has diverged and is serving an empty result
Intermittent, because the gateway routes to a different indexer each time. Same query, same subgraph, rows sometimes and nothing others, with no error anywhere. Query two indexers directly and compare.
- 4
There is genuinely no matching data
The only benign case, and it is indistinguishable from the other two in the response. That is why the status check is not optional.
Times this happened
- #5 one indexer diverged, serving [] intermittently with no error
The data is correct but months out of date
- 1
The subgraph broke on a contract upgrade
A proxy upgrade changes behaviour under a stable address while the mapping still expects the old ABI. The most common way a working subgraph breaks with no change on your side.
- 2
The source contract went quiet
The subgraph is at chain head, has no errors, and has been indexing the whole time. There is nothing to index, because the protocol you are watching migrated to a new contract. Check the contract address for activity before you blame the subgraph.
- 3
The chain halted, or the chain client behind the indexer froze
The most convincing lie in the ecosystem. graph-node reports synced, hasIndexingErrors is false, _meta returns a real block, and the gateway routes happily, because the indexer has genuinely indexed every block it can see. Compare _meta against a block explorer, not against the indexer.
- 4
Nobody maintains that subgraph any more
Find who published it in Graph Explorer and when it was last updated.
Times this happened
- #27 a proxy upgrade changed 11 of 13 event signatures, and the subgraph had been quietly wrong for months
- #7 the source contract migrated, the subgraph was healthy the whole time
- #13 the sole indexer reported 99.98% synced against a chain head frozen 85 hours
- #15 why a halted chain is indistinguishable from health in every tool
- #14 Moonbeam stopped producing blocks and nothing anywhere said so
- #25 a frozen chain client reporting synced while 3.68M blocks went unindexed
My query fails with "bad indexers" and a map of addresses
- 1
One or more operators have a failed copy of the deployment
Look for "no attestation: indexing_error" against an address. That copy has failed outright and is deterministic: it will not recover on its own and no amount of retrying helps. The operator has to intervene.
- 2
The reason inside the map is actually about your query
BadResponse(unattestable response: ...) usually means your query. Read the text inside the brackets rather than the wrapper around it. graph-node marks a class of errors unattestable, and the gateway wraps those so they arrive looking like an indexer fault.
- 3
Every operator is behind, unreachable, or returning a bare status
A mix of "too far behind", "no status: indexer not available" and bare BadResponse(400) across different addresses means the deployment is unserved rather than your query being wrong. When the failures disagree with each other, stop debugging your client.
- 4
Nothing is allocated at all
The error is then "no indexers found" rather than "bad indexers". Nothing was tried. That is a curation and allocation problem, not a serving one.
Times this happened
- #1 what every reason in that map means, decoded against gateway source
- #8 six indexers, four distinct failures, no healthy candidate anywhere
- #4 the reason inside the map turned out to be the query, not the indexers
- #26 the sole allocated indexer’s store was down, and there was no second candidate to fall back to
A brand new API key returns "auth error: API key not found"
- 1
The key has not propagated to the gateway yet
Reported as up to an hour, and regenerating the key does not help because the new one has the same problem. Nothing is wrong with the key. Wait, and do not build a workaround around it.
Times this happened
- #22 traced to the gateway auth path, read at a named commit
Building a subgraph
I published a subgraph and nothing is indexing it
- 1
It has no curation signal
Allocating to an unsignalled subgraph earns no indexing rewards, so no indexer will. Publishing is not a request to be served.
- 2
You signalled, but nobody has allocated yet
Signal makes a subgraph worth indexing. It does not compel anyone. Check allocations, not just signal.
Times this happened
- #21 the upgrade indexer itself was wedged on a store error and allocated to nothing
My subgraph takes days to sync
- 1
eth_calls inside handlers
A network round trip per call per event across the whole history. Different in kind from ordinary slowness, and worth fixing before anything else.
- 2
String IDs where bytes would do
Locale-aware UTF-8 comparison on every lookup. The docs measure up to 48% faster indexing from the change alone.
- 3
It is not actually stuck, and "99%" is not a real quantity
Sync progress is shown against an estimate. A deployment is at chain head or it is not, and one sitting at 99 percent for hours is usually still working through a dense range. Compare the indexed block against chain head directly rather than reading the percentage.
- 4
Mutable entities that never actually change
Every update maintains a block range on the previous version. Event-log entities can almost always be immutable.
Times this happened
- #10 reported repeatedly on different deployments, at chain head hours later every time
My graft is rejected, or a historical query is empty
- 1
The history was pruned
You cannot graft at a pruned height, and time travel queries are incompatible with prune: auto. Both fail quietly rather than loudly.
- 2
The schema change is not one grafting permits
Everything permitted is additive or a loosening. A new non-nullable field cannot be honest about inherited rows.
Pagination that worked for months started failing
- 1
An operator lowered their skip ceiling
The message is "The skip argument must be between 0 and 20000". GRAPH_GRAPHQL_MAX_SKIP is a per-operator graph-node setting, so your query was always at the mercy of a value you cannot see. You noticed the day somebody tightened it.
- 2
You are using skip at all
Order by id and filter on the last id you saw. Cursor pagination has no ceiling and does not degrade on a large subgraph, and it removes the dependency on every operator’s private configuration.
Times this happened
- #4 a 1TB subgraph, an operator at 20000, and a query unchanged for months
My subgraph will not deploy, with a connection error
- 1
The hosted service failed internally, and told you its private address
A 10.x.x.x, 172.16-31.x.x or 192.168.x.x address in the error is inside somebody else’s network. Your machine has never routed to it and never could. Nothing on your side is involved: not the manifest, not your CLI version, not your deploy key. Recognise the shape and stop debugging.
- 2
The manifest or the build is genuinely wrong
The distinguishing test is the address. A public host or a schema complaint is yours; a private address is not.
Times this happened
- #3 ECONNREFUSED on an RFC 1918 address, fixed by the operator forty minutes later
Studio keeps serving an old version of my subgraph
- 1
version/latest means the latest published version, not the newest deployment
Deploying is not publishing. Address the version label explicitly and drop version/ from the path. If that returns the new data, this was it.
Times this happened
- #2 v3.0.0 deployed, v2.1.0 served, fixed by naming the version in the URL
A field derived from IPFS is null and never fills in
- 1
The fetch missed once at index time and is never retried
graph-node does not retry ipfs.cat. A CID that was slow to propagate when the handler ran stays null for that entity forever, even though fetching the CID by hand now works perfectly. Test the CID directly: if it resolves and the entity is still null, the fetch is not being retried.
- 2
The content genuinely is not retrievable
Fetch the CID from more than one gateway. The two cases look identical in the data and only the direct fetch separates them.
Times this happened
- #6 both CIDs resolved by hand while one entity stayed null
Running an indexer
My allocation is open and healthy but earns nothing
- 1
The subgraph carries no curation signal
Indexing rewards depend on signal. A technically perfect allocation to an unsignalled subgraph earns nothing and looks entirely healthy.
- 2
The POI has gone stale
Past the staleness limit the allocation stops paying. Nothing is confiscated and no alarm sounds. Alert on POI age directly.
- 3
You are not eligible for indexing rewards
Since GIP-0079 rewards are gated on real gateway traffic reaching you on several separate days. The oracle reads gateway logs, not your metrics, so your own dashboards can be entirely green while the verdict is Unqualified. Check the oracle, not Prometheus.
Queries are being served but the revenue stopped
- 1
tap-agent has stopped, or RAV redemption is failing
Serving and settling are independent systems. Monitor unaggregated receipt value and the age of the oldest unredeemed RAV; neither shows up in query metrics.
- 2
The escrow behind the sender is exhausted
The receipts you hold are backed by nothing.
Times this happened
- #30 the sender’s aggregator was down, so no RAV could be signed and revenue stopped before serving did
I am allocated to a busy subgraph and serve none of its queries
- 1
Your cost model prices you above the field
Overpricing looks exactly like absence of demand from your side, which is why it persists for months. Compare query volume against allocated stake per subgraph.
- 2
Measured performance is putting the gateway off
Selection is a policy, and it is the gateway operator’s policy rather than the protocol’s.
Every query is rejected with "Invalid data_service: ... is not allowed for this indexer"
- 1
indexer-service does not recognise the data service paying you
A Horizon receipt names the data service it settles under. If the address in the error is 0xb2Bb92d0DE618878E438b55D5846cfecD9301105 on Arbitrum One, that is the SubgraphService proxy, which is the data service for subgraph queries: the gateway is paying you correctly and your side does not recognise the payer. Set subgraph_service_address under [blockchain] in the indexer-service config, and check receipts_verifier_address_v2 beside it. Both are required Horizon addresses.
- 2
The config predates the Horizon migration
The two addresses above did not exist before Horizon, so a config carried across from the pre-Horizon stack is missing them rather than wrong about them. Check whether they are absent rather than incorrect.
The gateway polls my cost endpoint every few seconds and sends me no queries
- 1
The queries are arriving and your indexer-service is rejecting them
Check this first, because it looks identical to never being selected and it is the one cause where the gateway is already doing its job. Receipt validation runs before the query reaches graph-node, so a rejected receipt is loud in the indexer-service log and invisible in every metric you own. Look for IndexerServiceError there before theorising about selection.
- 2
You are not allocated to the deployments carrying the demand
The polling proves you are discovered and reachable, so the fault is not the network path. But cost is fetched globally while queries route per deployment, and a gateway can only send you a query for something you are allocated to. Count allocations on deployments with real traffic, not allocations.
- 3
Your price loses the argmax
Selection is score divided by fee, so fee is a straight divisor and the strongest single lever. A measurably worse indexer takes first pick once its fee halves. Compare your model against what the deployment actually pays.
- 4
You are dropped from the candidate set before scoring
Two things remove you outright: roughly ten seconds or more further behind chainhead than the incumbent, and too little total stake. Neither shows up as an error and both look exactly like no demand.
Times this happened
- #17 selection measured against the compiled gateway crate, with the thresholds
I fixed the problem days ago and the oracle still says Unqualified
- 1
You are counting days since the repair, not qualifying days
The window is rolling and nothing resets it. The counter advances only on days a qualifying query actually landed, which needs the gateway to route to you first. Fixing your side enables that; it does not schedule it. Count days with traffic, not days since the fix.
- 2
Queries arrive and are rejected before they are served
A rejected receipt is a failed query, so it is a non-200 to the gateway and produces no qualifying day, while your own counter stays flat because it sits after receipt validation. Read the indexer-service log: this is indistinguishable from silence everywhere else.
- 3
The gateway is not routing to you
A flat query counter narrows it to this or the rejection above, and only the indexer-service log separates them. If nothing is being rejected either, the question becomes coverage, price and the candidate-set thresholds.
- 4
Traffic arrives but does not qualify
All three bars have to be met by the same query: 200, under the latency limit, and near chainhead. A deployment that has fallen behind answers every request successfully and produces no qualifying days at all.
- 5
You are testing against yourself
Queries you send your own endpoint never went through a gateway, so the oracle cannot see them. They will make your dashboards healthy and change nothing.
graph-node logs "Contract call reverted, reason: empty response" at volume
- 1
The archive node behind it is not serving historical state
Seen after moving Base to a base-reth-node storage v2 snapshot, on two independent operators. The chain client answers, so nothing reports unhealthy, but eth_call at a historical block comes back empty and the mapping records a revert that did not happen. Compare the same eth_call against a second provider at the same block.
Times this happened
- #11 open and unresolved, recorded early so nobody migrates blind
Every query is refused with "Received a receipt from a denylisted sender"
- 1
The sender’s aggregator is unreachable, so no RAV can be signed and your own tap-agent denied them
indexer-service is not judging the sender, it is reading a row your own tap-agent wrote, so check tap_horizon_denylist (scalar_tap_denylist on the legacy path) before you blame anybody. Then curl the URL mapped to that sender under [tap.sender_aggregator_endpoints]. A proxy erroring in front of the aggregator does not surface as an HTTP failure: tap-agent reports it as an invalid gRPC compression flag, and the flag number is the first byte of the proxy’s error body, 101 for "error code: NNN" and 60 for an HTML page. Ignore the flag and read the HTTP status at the end of the message.
- 2
tap-agent is failing to aggregate for some other reason, or is not running
The ceiling is the same whatever stopped aggregation, so a dead aggregator, a stopped agent and a failing redemption all arrive here looking identical. tap_sender_denied goes to 1 at the moment of denial and tap_sender_fee_tracker_grt_total climbs in a straight line to tap_max_fee_per_sender_grt_total beforehand; whether that line is climbing or flat says whether receipts are still arriving and only aggregation is broken.
- 3
The sender’s escrow no longer covers what you are holding
The other half of the deny condition, and it fires with no aggregation failure at all. Pending RAVs plus unaggregated fees exceeding the sender’s escrow balance denies them just the same, and indexer-service prints the identical error. Compare tap_sender_escrow_balance_grt_total against pending plus unaggregated before assuming the aggregator is at fault.
- 4
Invalid receipts have quietly accumulated
Invalid receipt fees count toward the same ceiling and never go down, so a slow trickle denies a sender eventually with nothing else wrong anywhere. tap_invalid_receipt_fees_grt_total is the one to watch, and it is the only one of these four that will not clear itself.
Times this happened
- #30 the mainnet aggregator serving a Cloudflare 526, reported as gRPC compression flag 101
Delegating
My rewards dropped and I changed nothing
- 1
GIP-0089 redirected a fifth of protocol issuance on 2026-08-31
Indexing rewards come from issuance. Trailing-average dashboards show this as a gradual drift rather than a step, which makes it easy to misread as your indexer.
- 2
Your indexer went over its delegation capacity
The dilution falls on everyone in the pool, not just late arrivals. An indexer can become a poor choice through other people’s deposits.
- 3
Your indexer changed its cut, or stopped allocating
The cut is a parameter they set and can move. Check parameter history and the allocated-versus-idle ratio.
Times this happened
- #23 GIP-0089 read off the contracts, including what had and had not activated
I undelegated and my GRT has not arrived
- 1
The thawing period has not elapsed
It earns nothing and cannot be moved during the wait. This is the mechanism, not a fault.
Curating
I signalled on a subgraph and have earned nothing
- 1
No indexer is allocated to it
Signal makes a subgraph attractive to index and compels nobody. No allocation means no fees, however much demand exists.
- 2
Nobody is querying it
Curators are paid from query fees, so the return is contingent on the forecast being right. This is the commonest outcome and it is silent.
- 3
The pool is crowded relative to the fees
Your return is your share of the pool. Heavy signal against modest fees divides the same income more ways.
Not here
If your symptom is missing, that is a gap worth reporting.
This index only holds failures somebody has written up. It is deliberately about the quiet ones: a process that crashes gets restarted by something you already have, and a subgraph that silently stopped three months ago does not.
It grows from graph-support, where the Night’s Watch triages this sort of thing in public and closes every issue with a stated outcome. Nothing here was invented: 25 of the entries below carry a link to the occasion somebody actually hit it. If yours is missing, filing it there is what puts it here, and you get an answer on the way.