Engineering

Search that scans every row

Airtable's API reference describes filterByFormula in one sentence that settles how fast your search box can ever be: the formula is evaluated for each record. No index sits behind it. A search feature wired to filterByFormula gets slower as the table grows, and the only lever you are given is to store fewer rows.

What a full scan costs

The API returns 100 records at a time. pageSize caps at 100 and defaults to 100, so a table sitting at the Team plan's 50,000-record ceiling takes 500 requests to walk end to end. Airtable allows 5 requests per second per base. Go past that and you get a 429 plus a 30-second wait before anything succeeds again. Five hundred requests at five per second is 100 seconds of wall clock for one pass over one table, and that assumes nothing else in your workspace is talking to the same base.

Team costs $24 per collaborator per month billed monthly, or $20 billed annually, and it includes 100,000 API calls per workspace per month. Spend 500 of them on each full scan and the month holds 200 scans. That is not 200 users each running a search. It is 200 scans for the whole month, for one feature. Caching papers over it for a while, and Airtable suggests exactly that in its own rate limit docs: if you anticipate a higher read volume, use a caching proxy. You are now running a cache so that a search box can work.

None of this is a flaw in Airtable. A spreadsheet-shaped database answering arbitrary predicates over user data has to look at the user data. The cost only becomes visible once a few thousand rows turn into fifty thousand, which is usually two years after someone made a sensible decision to start in Airtable.

The 16,000 character wall

Airtable accepts requests only when the URL is shorter than 16,000 characters, and the list-records page flags the risk twice: encoded formulas may cause your requests to exceed this limit. A formula you typed by hand never gets near it. A generated one does. Build an OR across whichever tags a user ticked, or feed in an ID list from a previous query, and the string grows with your data instead of with your code.

There is a documented way out. POST to /v0/{baseId}/{tableIdOrName}/listRecords and pass the parameters in the body rather than the query string. It works. It also turns your read path into a POST, which takes HTTP caching and every CDN layer in front of it off the table, and you tend to find out you need it on the day a customer selects more filters than you tested with.

What an index does instead

Postgres ships the alternative in core, no extension required. A GIN index keeps one entry per lexeme with a compressed list of the locations that match it. For a multi-word query it finds the first match, then uses the index to discard rows that lack the remaining words, so the work tracks how rare your search terms are rather than how many rows you own. GiST is the other option and the Postgres docs are direct about the trade: it is lossy, each document becomes a fixed-length signature of 124 bytes by default, and false matches force a trip back to the table. For text search, GIN is the preferred type.

On Supabase this is two statements. Add a tsvector column generated from the columns you want searchable, then create the GIN index on it. Postgres keeps the column current as rows change, and to_tsquery handles the stemming, so somebody typing eggs still matches a row that says egg. Ranking and prefix matching are already in the box, and neither costs an API call.

We went through this with Stay World Class when we moved them off Webflow and Xano onto Next.js, NestJS and Supabase. The data moved as a four-stage ETL that mapped every legacy ID to a PostgreSQL UUID, so no old link or report broke. Lighthouse went from 55.91 to 91 out of 100 and LCP came back 77% faster. Search stopped being something anyone brought up in a meeting.

When the scan is the right answer

A scan over 800 records is a scan over 800 records. If your base is small, or your search is really a filter on one single-select field, filterByFormula answers in a few hundred milliseconds and will go on doing that for years. Airtable's Free plan caps a base at 1,000 records and a workspace at 1,000 API calls per month, and at that size none of the arithmetic above means anything. Rewriting a working search box because a blog post made you nervous is a bad trade.

The signals that it is time are concrete. You are paging the same table over and over to answer a single query. You have started building a cache to hide the latency. Your generated filter is creeping toward the URL cap. Search ranking has become a product requirement and you have no way to express it. Until one of those is true, stay put and spend the week on something a customer asked you for.

Sources

Is your platform outgrowing its stack?

Book a call and we'll walk through how we'd approach your platform, with an estimate grounded in a working prototype.