Internet Search¶
The researcher searches the web with InternetSearch. It works out of the box
with a Gemini key, and you can add Serper or your own SearXNG. Agents use it on
their own; you only choose the providers.
How it works¶
Providers are tried in tiers, most authoritative first. The providers in a tier run in parallel, and the first tier that returns results answers the search:
| Tier | Providers | Used when |
|---|---|---|
| 1. Grounded | Google Search through Gemini | A Gemini key or Google Cloud project is set |
| 2. Web | Serper and SearXNG | Tier 1 is not configured or found nothing |
| 3. Scholarly | arXiv and Crossref | No web provider found anything |
| 4. Last resort | Wikipedia | Only with RESEARCH_ALLOW_WIKIPEDIA_FALLBACK=true, or when every web provider is excluded |
Within a tier, results are de-duplicated by URL and ranked by the provider's weight (Google 1.4, Serper 1.2, SearXNG 1.1, arXiv 0.9, Crossref 0.8, Wikipedia 0.6), plus how many query words they contain, plus a small bonus for PDFs.
A provider that fails or times out is skipped, and the search moves on. When no web provider could answer at all, the researcher is told the search was unavailable rather than handed an empty list it might read as "nothing exists".
Providers¶
| Provider | Setup | Notes |
|---|---|---|
| Google grounded search | GEMINI_API_KEY (or GOOGLE_GENAI_API_KEY), or GOOGLE_CLOUD_PROJECT with Application Default Credentials |
The best results. GEMINI_GROUNDING_API_KEY and GEMINI_GROUNDING_MODEL (default gemini-2.5-flash) override the key and model; GOOGLE_CLOUD_LOCATION defaults to global. |
| Serper | SERPER_API_KEY from serper.dev |
Google results through a paid API. |
| SearXNG | SEARXNG_INSTANCE_URL (default http://localhost:8080) |
Self-hosted metasearch with no API keys. If you haven't set the URL and nothing is running there, it is skipped quietly. |
| arXiv, Crossref | None | Free public APIs for papers and DOIs. |
| Wikipedia | RESEARCH_ALLOW_WIKIPEDIA_FALLBACK=true |
Off unless you turn it on (see above). |
To run SearXNG locally, with its JSON output turned on:
Settings¶
GEMINI_API_KEY=AIza... # Google grounded search
SERPER_API_KEY=... # optional
SEARXNG_INSTANCE_URL=http://searxng:8080 # optional
RESEARCH_SEARCH_TIMEOUT_SECONDS=15 # per provider
RESEARCH_GROUNDED_TIMEOUT_SECONDS=45 # Google grounding
RESEARCH_PDF_TIMEOUT_SECONDS=90
RESEARCH_PDF_MAX_RETRIES=3
RESEARCH_USER_AGENT="Acme Research research@acme.example" # required by registers such as SEC EDGAR
Every setting is in Configuration.
Skipping providers per call¶
When you call search yourself, skip providers that don't fit the task. A skipped provider makes no network call:
from jarviscore.search import InternetSearch
async with InternetSearch() as search:
results = await search.search(
"latest SEC filings for AAPL",
max_results=10,
exclude_providers={"arxiv", "crossref", "wikipedia"},
)
Provider names: google_grounded, serper, searxng, arxiv, crossref,
wikipedia.
Reading pages¶
The researcher reads the pages it finds over plain HTTP, PDFs included. A page
that needs JavaScript is opened in a headless browser when the browser extra is
installed (pip install "jarviscore-framework[browser]"), or in your own
browser with BROWSER_CONTROL_URL. For pages that have to be used rather than
read, see Browser Automation.
A page that fails to load carries the HTTP status_code; one the site refuses
with 401 or 403 is marked access: protected. The agent decides what to do
next; the runtime does not retry on its behalf.
Which providers to use¶
| You want | Use |
|---|---|
| To get started | GEMINI_API_KEY alone |
| Google results without Gemini grounding | SERPER_API_KEY, and exclude google_grounded |
| No third-party search API | Self-hosted SearXNG, and exclude google_grounded and serper |
| Academic sources | arXiv and Crossref answer when the web finds nothing. To search them directly, exclude the web providers |
| News and markets | Google grounding and Serper, excluding the scholarly providers |