investdatar

investdatar is an R package for retrieving, standardizing, and locally synchronizing investment-related datasets in R. It acts as a data access and local storage layer for financial datasets, with a consistent provider-facing workflow for:

Supported providers currently include:

Installation

# CRAN release, when available:
# install.packages("investdatar")

# Development version:
# install.packages("remotes")
# remotes::install_github("OliverLDS/investdatar")
install.packages("investdatar", repos = c('https://oliverlds.r-universe.dev', 'https://cloud.r-project.org'))

Development versions are available through GitHub. Use CRAN releases when available.

Configuration

Users should define INVESTDATAR_CONFIG in their .Renviron file. It must point to a YAML file. A minimal example is shipped with the package at inst/extdata/investdatar_config_example.yaml. Credentials such as FRED_API_KEY, ALPHAVANTAGE_API_KEY, EIA_API_KEY, BLS_API_KEY, BEA_API_KEY, CENSUS_API_KEY, and SEC_USER_AGENT should also be stored in .Renviron when needed.

INVESTDATAR_CONFIG=/absolute/path/to/investdatar_config.yaml
FRED_API_KEY=your_fred_key
ALPHAVANTAGE_API_KEY=your_alphavantage_key
EIA_API_KEY=your_eia_key
BLS_API_KEY=your_optional_bls_key
BEA_API_KEY=your_bea_key
CENSUS_API_KEY=your_census_key
SEC_USER_AGENT=Your Name your_email@example.com

The YAML file is intended for local storage paths and source-specific metadata. Some providers require API keys, while others such as Yahoo Finance do not. OKX candle sync also falls back to package defaults when no config is passed; set OKX_API_KEY, OKX_SECRET_KEY, and OKX_PASSPHRASE in .Renviron if you want authenticated OKX access.

Minimal example:

FRED:
  data_path: /absolute/path/to/fred_data
  registry_file: /absolute/path/to/fred_macro_series_registry.json

WorldBank:
  data_path: /absolute/path/to/world_bank_data
  registry_file: /absolute/path/to/world_bank_series_registry.json

Treasury:
  data_path: /absolute/path/to/treasury_data

CFTC:
  data_path: /absolute/path/to/cftc_data
  registry_file: /absolute/path/to/cftc_cot_registry.json

FiscalData:
  data_path: /absolute/path/to/fiscal_data
  registry_file: /absolute/path/to/fiscaldata_registry.json

EIA:
  data_path: /absolute/path/to/eia_data
  registry_file: /absolute/path/to/eia_series_registry.json

BLS:
  data_path: /absolute/path/to/bls_data
  registry_file: /absolute/path/to/bls_series_registry.json

BEA:
  data_path: /absolute/path/to/bea_data
  registry_file: /absolute/path/to/bea_series_registry.json

Census:
  data_path: /absolute/path/to/census_data
  registry_file: /absolute/path/to/census_series_registry.json

SEC:
  data_path: /absolute/path/to/sec_data
  registry_file: /absolute/path/to/sec_company_registry.json
  frames_registry_file: /absolute/path/to/sec_frames_registry.json

SDMX:
  data_path: /absolute/path/to/sdmx_data
  registry_file: /absolute/path/to/sdmx_series_registry.json

RSS:
  data_path: /absolute/path/to/rss_data
  registry_file: /absolute/path/to/rss_feed_registry.json
  # Optional request-scoped overrides for runtimes with an incomplete CA bundle.
  # feed_ca_bundles:
  #   cftc_press_releases: /absolute/path/to/current-ca-bundle.pem

Crypto:
  data_path: /absolute/path/to/crypto_data
  derivatives_registry_file: /absolute/path/to/crypto_derivatives_registry.json
  # OKX local files are stored under /absolute/path/to/crypto_data/okx
  # Binance local files should be stored under /absolute/path/to/crypto_data/binance

iShare:
  data_path: /absolute/path/to/ishare_data
  registry_file: /absolute/path/to/ishare_ticker_registry.json
  holdings_tickers: [DYNF, THRO, BAI, BDYN, BDVL]

YahooFinance:
  data_path: /absolute/path/to/yahoo_finance_data
  registry_file: /absolute/path/to/YahooFinance_ticker_registry.json

AlphaVantage:
  data_path: /absolute/path/to/alphavantage_data
  registry_file: /absolute/path/to/alphavantage_series_registry.json

Relative paths are also supported and are resolved relative to the config file location.

First Run

Start from the shipped example, adjust the local paths, then point INVESTDATAR_CONFIG at your copy.

example_cfg <- system.file("extdata", "investdatar_config_example.yaml", package = "investdatar")
example_cfg

Basic Usage

library(investdatar)

cfg <- get_investdatar_config()

fred_dt <- get_source_data_fred("DGS10")
fred_sync <- sync_local_fred_data("DGS10")
fred_local <- get_local_FRED_data("DGS10")

wb_dt <- get_source_data_wbstats("NY.GDP.MKTP.CD", country = "US")
wb_sync <- sync_local_wbstats_data("NY.GDP.MKTP.CD", "US")
wb_local <- get_local_wbstats_data("NY.GDP.MKTP.CD", "US")

treasury_dt <- get_source_data_treasury_rates("par_yield_curve", years = 2026)
treasury_sync <- sync_local_treasury_rates("par_yield_curve")
treasury_local <- get_local_treasury_rates("par_yield_curve")

cftc_dt <- get_source_data_cftc_cot("futures_only", market_codes = "020601")
cftc_sync <- sync_all_cftc_cot_registry_data()
cftc_local <- get_local_cftc_cot("tff_futures_only")

fiscal_sync <- sync_all_fiscaldata_registry_data()
debt_local <- get_local_fiscaldata("debt_to_penny")

eia_sync <- sync_all_eia_registry_data()
crude_stocks <- get_local_eia_data("PET.WCESTUS1.W")

sec_submissions_sync <- sync_all_sec_submissions_registry_data()
sec_facts_sync <- sync_all_sec_companyfacts_registry_data()

sdmx_sync <- sync_all_sdmx_registry_data()
ecb_fx <- get_local_sdmx_data("ecb_usd_eur_daily")

derivatives_sync <- sync_all_crypto_derivatives_registry_data()
btc_funding <- get_local_crypto_derivatives("binance", "funding_rate", "BTCUSDT", "funding")

rss_dt <- get_source_data_rss("atlfed_gdpnow", "https://www.atlantafed.org/rss/GDPNow", parser = "gdpnow")
rss_sync <- sync_local_rss_data("atlfed_gdpnow", "https://www.atlantafed.org/rss/GDPNow", parser = "gdpnow")
rss_local <- get_local_rss_data("atlfed_gdpnow")

ishare_local <- get_local_ishare_data("IVV")
ishare_holdings_sync <- sync_all_ishare_registry_holdings()
ishare_holdings_local <- get_local_ishare_holdings("DYNF")

okx_local <- get_local_okx_candle("BTC-USDT-SWAP", "4H")

yahoo_dt <- fetch_quantmod_OHLC("SPY", from = "2024-01-01", to = "2024-12-31")
yahoo_sync <- sync_all_yahoofinance_registry_data(from = "2024-01-01", to = "2024-12-31")

specs <- list_source_specs()

prompt_txt <- describe_fred_data("DGS10")

Minimal local-sync workflow:

library(investdatar)

cfg <- load_investdatar_config(Sys.getenv("INVESTDATAR_CONFIG"))

fred_sync <- sync_local_fred_data("DGS10")
fred_local <- get_local_FRED_data("DGS10")
fred_meta <- get_local_data_meta(fred_sync$file_path)

For spec-driven local access, the current local-reader functions map to source specs as follows:

Local path conventions for other market-data specs:

Current local sync helpers include:

For large candle repair workflows, prefer batch repair helpers that fetch all missing pages or windows in memory and write the local .rds file once:

OKX and Binance candle readers, sync functions, and repair helpers also accept storage = "monthly". This migrates a monolithic cache into YYYY-MM.rds partitions and rewrites only months touched by an upsert; bounded reads load only relevant partitions.

Yahoo Finance registry batch sync is also available through sync_all_yahoofinance_registry_data(). It reads tickers from the configured YahooFinance.registry_file and synchronizes each one via quantmod. Each symbol receives bounded retries with exponential backoff; incomplete OHLC windows are reported as errors and are not upserted into the local cache. Completeness requires finite open, high, low, and close values. End-of-window coverage allows a seven-calendar-day grace period for weekends and market holidays. Start-of-window coverage and the minimum weekday-row check apply only when valid local bars already establish the instrument's history, so a newly listed instrument is not rejected merely for lacking earlier data. Isolated non-finite bars are dropped rather than being allowed to overwrite a valid local bar; materially short windows are still rejected. The shipped 72-symbol seed registry is inst/extdata/config/YahooFinance_ticker_registry.json; copy it into the configured runtime path when initializing a local registry. Prefer the deterministic bootstrap below: it creates an absent runtime registry from the tracked seed and refuses to overwrite an existing one. Validate an existing registry before scheduled syncs so required fallback declarations cannot drift silently across machines.

bootstrap_yahoofinance_registry()
validate_yahoofinance_registry()

When validation fails, restore the required fallback entries from the tracked seed. Alternatively, first back up the existing runtime JSON file, remove it, and run bootstrap_yahoofinance_registry() to recreate it. The default Yahoo registry batch sync performs this validation before making provider requests.

For the two configured Eastmoney-backed instruments, CNH=X and 000300.SS, the registry may explicitly opt into a bounded degraded_cache outcome with degraded_cache_enabled: true and degraded_cache_max_staleness_days. This outcome is used only when both fresh providers fail and the local completed OHLC cache is finite, within its calendar cutoff and staleness limit, and has no unresolved integrity finding. It never writes or repairs cache rows. All other symbols, and any missing, invalid, stale, or integrity-failed cache, remain hard failures.

For a known Yahoo-only failure, a registry row can opt into an explicit, provenance-preserving fallback rather than silently substituting data:

{
  "yahoo_finance_ticker": "000300.SS",
  "fallback_source": "eastmoney",
  "fallback_ticker": "1.000300"
}

Fallback bars retain source = "eastmoney", and batch summaries identify fetch_method = "eastmoney_fallback", fetch_attempts, and the failed primary request in primary_error. Before an external fallback, Yahoo failures also try Yahoo's chart-range endpoint and report fetch_method = "yahoo_chart_range_fallback" when that same-source recovery succeeds. The external fallback is limited to declared rows; it never replaces a finite local Yahoo bar and all other symbols continue to use Yahoo through quantmod.

Eastmoney's historical-kline host can intermittently close a connection before returning HTTP headers. Its fallback uses five TLS-verified, bounded transport attempts with source-compatible headers. If those attempts are exhausted, Investdatar returns a typed transient error recommending a retry after 60 seconds; the ticker remains a failed batch entry rather than being reported as successfully synchronized.

Some package-maintained Yahoo entries are marked required: true in the seed. An existing runtime registry must retain those ticker rows, in addition to the declared fallback mappings; validation fails before sync if they are missing. This keeps maintained dependency data in the normal Yahoo sync workflow while still allowing additional runtime-only ticker rows.

Instrument Catalog

get_instrument_catalog() returns a package-owned, versioned catalog of factual instrument metadata for downstream consumers. It currently includes SPY, TLT, GLD, EUR/USD, BTC/USD, EEM, HYG, USO, plus the Yahoo fallback-backed USD/CNH and CSI 300 mappings, and a separate six-instrument 4-hour OKX perpetual-swap subset. The catalog includes stable instrument_id and canonical_symbol fields, market taxonomy, quote currency, market calendar, provider identifiers, intended primary routing, fallback sources, intervals, and active status. It does not include editorial, ranking, simulation, strategy, or execution fields. Batch 3 adds Treasury yields and U.S., European, Japanese, and Hong Kong indices. Batch 4 adds CME and ICE continuous futures for Treasury notes, crude oil, gold, silver, and copper.

The catalog accepts legacy schema 1.0.0 records and provides schema 1.1.0 normalization through normalize_instrument_catalog(). In schema 1.1.0, market_calendar means only the completed-bar session calendar; legacy MIC-like values such as XNYS are calendar aliases, not venue metadata. primary_listing_mic is a separate nullable venue field, quote_unit is required, and audit_profile explicitly controls tolerance, volume semantics, and completion rules. The dedicated ICE_DXY calendar treats DXY as a weekday index with only observed New Year and Christmas holidays; it is not an equity or generic FX calendar. primary_listing_mic remains null for indices, yields, and continuous futures because it is venue provenance, not a completed-bar calendar. Yahoo futures symbols are provider-defined continuous series, not deliverable contracts; their roll treatment and volume semantics are explicit in continuous_contract. DX-Y.NYB is mapped to the ICE U.S. Dollar Index (DXY) with Yahoo retained only as its provider identifier; Yahoo chart metadata is not treated as the benchmark authority.

The read-only Yahoo overlap audit distinguishes provider availability from cache integrity. A window with no usable Yahoo rows emits one availability finding and does not infer missing or invalid cache data. For 000300.SS and CNH=X, an Eastmoney comparison is attempted under fallback_corroboration = "on_issue"; a successful comparison can expose actual integrity differences, while failure of both sources produces audit_incomplete_provider_unavailable with both provider errors. Normal synchronization remains strict and still reports a failed Yahoo/Eastmoney chain as a provider failure.

catalog <- get_instrument_catalog()
validate_instrument_catalog()

# Refined schema 1.1.0 view; provider identifiers and sync behavior are unchanged.
catalog_1_1 <- normalize_instrument_catalog()

# Provider-neutral 4H linear USDT-margined perpetual swaps.
okx_perpetuals <- get_okx_perpetual_catalog()
validate_okx_perpetual_catalog(
  local_path = "~/Documents/2025/_2025-06-17_Crypto_Data/okx"
)

canonical_symbol is a consumer identity, not a provider query. Current Yahoo local-cache access and synchronization resolve through provider_identifiers$yahoo; it must not be passed directly to get_completed_local_quantmod_OHLC(). primary_source is intended routing metadata. Actual row-level provenance is the source value recorded by the synchronization result. The Yahoo Finance registry remains the sync authority. Catalog validation requires every Yahoo-backed instrument to exist in that registry and checks catalog fallback projections against its configured fallback declarations. Every fallback object has exactly provider and symbol fields; provider/symbol pairs are unique and ordered lexicographically.

get_okx_perpetual_catalog() returns the 4-hour OKX perpetual catalog entries, currently BTC, ETH, SOL, BNB, XRP, and DOGE against USDT. Their provider_identifiers$okx values resolve local cache filenames; their canonical_symbol values are consumer identities. Contract metadata is the current OKX public-instruments specification: contract_size, quantity_step, settlement_currency, and contract_structure = "linear_usdt_margined_perpetual". Supplying local_path to validate_okx_perpetual_catalog() requires every active entry to have a non-empty cache with a finite completed OHLC row. Normal OKX ingestion continues to exclude unconfirmed (confirm != 1) candles.

Daily Yahoo OHLC cache rows dated on the current UTC date are provisional even when open, high, low, close, and volume are finite. Source timestamps and cache freshness indicate retrieval timing, not that a daily bar is final. Raw reads retain those rows; use get_completed_local_quantmod_OHLC() with an explicit UTC as_of timestamp for analysis requiring completed daily bars. The next overlap sync upserts the finalized same-date row.

World Bank registry batch sync is available through sync_all_wbstats_registry_data(). It reads indicator definitions from the configured WorldBank.registry_file and synchronizes each registered indicator + country + freq series. If country is blank in the registry, the sync falls back to the package default World Bank scope, which is countries_only.

Treasury raw-rate batch sync is available through sync_all_treasury_rates(). It synchronizes the five built-in Treasury datasets into the configured Treasury.data_path:

CFTC Commitments of Traders batch sync is available through sync_all_cftc_cot_registry_data(). The registry pins the official TFF, Disaggregated, and Legacy futures-only and combined datasets and can optionally restrict downloads to selected CFTC contract-market codes. Local synchronization uses a two-week overlap and keyed upserts so routine runs retrieve only recent report weeks while retaining corrected values.

Treasury Fiscal Data batch sync is available through sync_all_fiscaldata_registry_data(). The shipped registry covers Debt to the Penny, the Daily Treasury Statement Operating Cash Balance, auctions, monthly receipts and outlays, interest expense, and Treasury securities outstanding. Each entry declares its endpoint and key columns, allowing heterogeneous Treasury tables to retain their source fields while sharing pagination, incremental synchronization, metadata, and run-log behavior.

EIA registry batch sync is available through sync_all_eia_registry_data(). The initial registry tracks six weekly physical-market fundamentals covering petroleum inventories, crude production and refinery inputs, and Lower-48 natural-gas storage. Set EIA_API_KEY in .Renviron; routine syncs overlap the latest local month and upsert revised observations.

SEC EDGAR uses one company registry for two independent local datasets. sync_all_sec_submissions_registry_data() stores filing events keyed by CIK and accession number, including historical submission files on first sync. sync_all_sec_companyfacts_registry_data() stores XBRL facts in long form while retaining taxonomy, unit, reporting context, accession, and amendment details. Set SEC_USER_AGENT to an identifiable contact before making SEC requests. Cross-company XBRL Frames can be cached with sync_local_sec_frame() or an explicit Frames registry. sync_sec_filing_documents() downloads only selected primary documents from cached submissions, while sync_local_sec_bulk_archive() keeps the SEC nightly bulk ZIPs opt-in.

SDMX batch sync is available through sync_all_sdmx_registry_data(). Registry entries declare the provider, dataflow, key, CSV format, observation columns, and dimensions; the local canonical fields are stored alongside the original provider columns. The shipped registry includes ECB exchange and policy rates, BIS policy rates, Eurostat HICP, IMF DataMapper macro indicators, and an OECD composite-leading-indicator seed using the official SDMX REST v1 endpoint.

Crypto derivatives batch sync is available through sync_all_crypto_derivatives_registry_data(). The shipped registry tracks BTC and ETH funding, open interest, mark/index prices, basis, and Binance long-short ratios. Public websocket liquidation events can be upserted with sync_local_crypto_liquidations(); private account force-order history is not mislabeled as market-wide liquidation data.

BLS, BEA, and Census provide selective direct-agency registry workflows through sync_all_bls_registry_data(), sync_all_bea_registry_data(), and sync_all_census_registry_data(). The seeds focus on labor-market series, state GDP/income panels, and advance retail sales where the direct APIs expose useful source dimensions.

RSS feed registry batch sync is available through sync_all_rss_registry_data(). It reads feed metadata from the configured RSS.registry_file and synchronizes each configured feed into a local .rds table. If the runtime's default certificate store cannot verify one feed, add that feed under RSS.feed_ca_bundles or set its registry ca_bundle field. The override applies only to that feed request and does not disable TLS verification. Registry-level ca_bundle values take precedence over config.

The shipped example registry includes Atlanta Fed, SEC, Federal Reserve, and CFTC seeds:

[
  {
    "feed_id": "atlfed_gdpnow",
    "provider": "atlanta_fed",
    "url": "https://www.atlantafed.org/rss/GDPNow",
    "type": "macro_narrative",
    "parser": "gdpnow",
    "main_group": "us_growth_nowcast",
    "active": true
  },
  {
    "feed_id": "sec_press_releases",
    "provider": "sec",
    "url": "https://www.sec.gov/news/pressreleases.rss",
    "type": "regulatory_press_release",
    "parser": "plain",
    "main_group": "us_regulation",
    "active": true
  },
  {
    "feed_id": "fed_press_all",
    "provider": "federal_reserve",
    "url": "https://www.federalreserve.gov/feeds/press_all.xml",
    "type": "central_bank_press_release",
    "parser": "plain",
    "main_group": "us_monetary_policy",
    "active": true
  },
  {
    "feed_id": "cftc_press_releases",
    "provider": "cftc",
    "url": "https://www.cftc.gov/RSS/RSSGP/rssgp.xml",
    "type": "regulatory_press_release",
    "parser": "plain",
    "main_group": "us_derivatives_regulation",
    "active": true
  }
]

For iShares holdings, sync_all_ishare_registry_holdings() no longer syncs the entire iShares registry by default. It reads iShare.holdings_tickers from the package config and, unless you override it, tracks only:

standardize_fund_holdings() defines a provider-neutral long holdings contract; get_local_ishare_holdings_standardized() converts existing iShares caches without changing their backward-compatible file layout.

Alpha Vantage supports local readers, incremental full/compact sync, a registry batch workflow, sidecar metadata, and run logs in the same pattern as other market providers.

Notes

Local Verification

This repository includes a local-library verification workflow so package tests do not depend on whatever happens to be installed in the global R library.

scripts/install-local-lib.sh
scripts/verify-local.sh

By default, both scripts use INVESTDATAR_LOCAL_LIB=/tmp/investdatar-r-lib.