View source for OSINT Stack Architecture

= OSINT Stack Architecture =

Personal open-source intelligence collection and analysis system running on the VPS. Designed for Hudson Valley regional monitoring with a focus on ICE/immigration enforcement, government contracts, aviation anomalies, and civic infrastructure.

Last updated: 2026-03-05.

== Overview ==

The stack follows a hub-and-spoke architecture:

* '''Domain collectors''' each monitor a narrow slice of open data and push signals to a central hub
* '''[[Anomalywatch]]''' aggregates all signals, scores them, detects correlations, and maintains active investigations
* '''[[Entityhub]]''' maintains a knowledge graph of entities (people, orgs, aircraft, locations) with cross-app mention tracking
* '''Intel Dashboard''' (intel.tools.ejfox.com) provides a unified ops view

All apps run as [[Smallweb]] Deno apps under a shared reverse proxy on port 7777. Internal routing uses <code>http://localhost:7777</code> + <code>Host: appname.tools.ejfox.com</code> header.

== Signal Flow ==

<pre>
Domain Collector --> POST /api/report-alert --> Anomalywatch
                                                      |
                                              score + deduplicate
                                                      |
                                           Investigations + Correlations
                                                      |
                                             Intel Dashboard
                                                      |
                                            Briefing (claude-briefing.sh)
</pre>

Entity resolution runs in parallel:

<pre>
Collector --> GET /api/resolve?q=NAME&create=true --> Entityhub
          --> POST /api/entity/:id/mention          --> Entityhub
</pre>

== Domain Collectors ==

=== Aviation / Maritime / Movement ===

; [[Skywatch]] (skywatch.tools.ejfox.com)
: FAA ADSB data via adsb.lol. Monitors Hudson Valley airspace for military flights, ICE Air Operations, surveillance patterns, unidentified aircraft. Flags: military callsigns, known ICE prefixes (ICE+mixed alphanumeric), low-altitude surveillance loops.
: DB: <code>/opt/docker/smallweb/data/skywatch/data/skywatch.db</code>

; [[AIS Collector]] / Riverwatch
: AIS vessel tracking via aisstream.io WebSocket. Monitors Hudson River and surrounding waters. Flags: military vessels, law enforcement, tankers, unusual speeds (>25kt for unknown type, >15kt for known).
: PM2 process: <code>ais-collector</code>
: DB: <code>~/scripts/scripts/ais-collector/riverwatch.db</code>
: See [[Hudson River AIS Intelligence Brief]] for vessel speed thresholds, traffic context, and flagging methodology.

; [[Masintwatch]] (masintwatch.tools.ejfox.com)
: 9 MASINT (measurement and signature intelligence) sources: FAA NOTAMs, USGS streamflow, NRC nuclear events, EPA air quality, NWS/NEXRAD radar, USCG notices to mariners, USGS seismic, NASA FIRMS fire detection, NYISO grid load. Monitors for environmental anomalies near sensitive HV sites (Stewart ANG, Indian Point, Chester ICE facility, West Point, Algonquin Pipeline).

=== Government / Financial ===

; [[Contractwatch]] (contractwatch.tools.ejfox.com)
: USASpending.gov federal contracts. Tags contracts with <code>geo_region=hudson_valley</code> for 9 NY counties (Orange, Ulster, Dutchess, Sullivan, Greene, Columbia, Putnam, Rockland, Westchester). Flags DHS/CBP/ICE awards.
: DB: <code>/opt/docker/smallweb/data/contractwatch/data/contractwatch.db</code>
: Key columns: <code>recipient_name</code>, <code>award_amount</code>, <code>geo_region</code>

; [[Donorwatch]] (donorwatch.tools.ejfox.com)
: FEC campaign finance. Monitors watched donors via name-pattern substring matching. Reference implementation for anomalywatch reporting + entityhub resolution patterns.
: DB: <code>/opt/docker/smallweb/data/donorwatch/data/donorwatch.db</code>
: Key table: <code>watched_donors</code> (name_pattern passed to FEC API)

; [[Filingwatch]] (filingwatch.tools.ejfox.com)
: SEC filings and regulatory documents.

; [[990watch]]
: IRS Form 990 nonprofit data. Monitors watched nonprofits for compensation anomalies, revenue changes, new filings. Alert dedup: pre-insert SELECT check prevents repeat alerts for same finding (NRA $5.57M, Candid $1.38M previously fired every scan cycle).
: Key table: <code>watched_nonprofits</code>

=== Location / Property ===

; [[Countywatch]] (countywatch.tools.ejfox.com)
: County government RSS feeds (Orange County Legislature, Planning Board, ZBA), Chester cit-e.net planning docs, local news. ICE keyword monitoring on meeting docs. ~30 feeds.
: Key tables: <code>news</code> (column: <code>first_seen</code>), <code>meetings</code>

; [[Egoscan]] (egoscan.tools.ejfox.com)
: Domain/WHOIS monitoring and username enumeration. Alerts on new domain registrations, expiring domains (30 days or fewer), WHOIS changes, username appearances on new platforms.
: Scan endpoints: <code>/api/scan/core</code>, <code>/api/scan/quick</code>

; [[Changewatch]] (changewatch.tools.ejfox.com)
: Web page change detection. Monitors government and organizational URLs for content changes. Each watched URL can optionally link to an entityhub entity.

=== Background Data (Socrata batch feeds) ===

These sources load bulk public records. Signals are tagged <code>socrata:*</code> or <code>openfda:*</code> and '''filtered from Top Signals''' in the intel dashboard and briefing -- they appear in a collapsed "Background Data" section instead.

* Restaurant inspections (NYC DOH)
* Remediation sites
* Building permits
* Liquor licenses
* Hate crime reports
* FDA recalls

=== Security / Infrastructure ===

; [[Honeypot]]
: HTTP honeypot logging all unauthorized access attempts. File-based logging (JSONL at <code>data/honeypot-log.jsonl</code>). 4,388+ attacks logged. Anomalywatch integration active.
: Log rotation: archives >5MB, keeps last 7 days.

== Central Hub: Anomalywatch ==

<code>/opt/docker/smallweb/data/anomalywatch/main.ts</code> (~2,100 lines)

=== Signal Scoring ===

Signals arrive via <code>POST /api/report-alert</code>. The scoring pipeline:

# Source weight multiplied by base score
# Geographic relevance bonus (hudson_valley tagged signals get a boost)
# Recency decay
# Final score stored as <code>final_score</code>

Source weight taxonomy:
* '''intelligence''' sources (skywatch, donorwatch, contractwatch, changewatch, filingwatch, egoscan): full weight
* '''background_data''' (socrata:*, openfda:*): low weight, filtered from top signals
* '''system''' (osint-health, honeypot): excluded from briefings

=== Deduplication ===

Two-layer dedup (fires in order):
# '''Exact title + same source within 24h''' -- deduplicated, no new signal row
# '''Word overlap >= 0.7 within 2h''' -- deduplicated

=== Correlations ===

Computes Pearson r on '''z-score vectors''' (not raw counts). Only flags correlations where '''both''' series show z > 1.5 in the same window. This prevents batch-loaded Socrata sources from correlating with everything (they grow on predictable daily schedules, so raw count r is approximately 1.0 but z-score r is approximately 0).

=== Active Investigations ===

{| class="wikitable"
! ID !! Name !! Signal Types !! Focus
|-
| inv-mm9cq6jy-v6qi || [[Chester ICE Facility|Chester ICE Detention Complex]] || aviation, government, crime || hudson_valley / Chester NY
|-
| inv-mm9cq6n7-i1n3 || Stewart ANG Regional Activity || aviation || hudson_valley / Stewart Airport
|-
| inv-mm9cq6or-l8pt || Hudson Valley Federal Contracts || government || hudson_valley
|}

New signals matching investigation keywords/types/geography automatically link to the <code>investigation_signals</code> table.

=== Key API Endpoints ===

* <code>GET /api/signals?hours=24&min_score=2.0&limit=20</code> -- recent high-score signals
* <code>GET /api/intelligence/summary</code> -- aggregate stats (signals, anomalies counts)
* <code>GET /api/investigations</code> -- active investigations
* <code>GET /api/correlations</code> -- cross-source correlation matrix
* <code>GET /api/query?geo_region=hudson_valley&since=30d&sources=skywatch,contractwatch</code> -- cross-signal join query
* <code>POST /api/report-alert</code> -- collector ingestion endpoint

== Entity Graph: Entityhub ==

<code>/opt/docker/smallweb/data/entityhub/main.ts</code>

Central knowledge graph. All collectors resolve named entities before recording signals.

=== Schema ===

* <code>entities</code> -- canonical_name, entity_type (person/organization/aircraft/location/vessel), external IDs
* <code>entity_external_ids</code> -- Wikidata QIDs, FAA reg numbers, FEC IDs, etc.
* <code>entity_mentions</code> -- app_name, entity_id, context, timestamp
* <code>intel_alerts</code> -- elevated alerts derived from mention patterns

=== Entity Resolution Pattern ===

Used by donorwatch (reference implementation), 990watch, AIS collector, contractwatch:

<pre>
GET /api/resolve?q=NAME&create=true&type=TYPE
--> returns entity_id (existing or newly created)

POST /api/entity/:id/mention
--> { app_name, context, metadata }
</pre>

Auto-enrichment fires on <code>create=true</code> for person/org/location: non-blocking Wikidata lookup with 500ms delay.

=== Statistics (2026-03-02 post-cleanup) ===

* 4,797 entities (down from 8,001 after NER noise removal)
* Types: aircraft, organization, person, location, vessel
* 3,163 unknown-type and 3,765 concept-type entities removed (NER noise from scrapbook bulk ingestion)

=== Smallweb Permissions ===

Entityhub DB owned by <code>debian:debian</code> -- safe to query directly from CLI. Sibling app DBs owned by www-data may create ownership issues on WAL files if queried as debian.

== Intel Dashboard ==

<code>intel.tools.ejfox.com</code> -- <code>/opt/docker/smallweb/data/intel/main.ts</code>

Aggregates data from anomalywatch, skywatch, entityhub, countywatch DBs directly (requires <code>"admin": true</code> in smallweb.json for cross-app DB access).

Layout: header + stats bar + 3-column main (signals | sky activity | investigations+alerts) + 3-column bottom (health | county news | correlations).

Filters applied:
* Excludes osint-health signals from top signals
* Excludes socrata:* and openfda:* from top signals
* Shows background data in separate collapsed section

== Briefing System ==

<code>/home/debian/scripts/claude-briefing.sh</code>

Generates a markdown briefing that Claude reads at session start. Pulls:
* Top intelligence signals (min_score >= 2.0, excludes socrata/openfda)
* Active investigation status
* Skywatch daily stats + military activity
* Countywatch recent news
* Kanban board status
* Background data section (socrata/openfda, collapsed)

Runs every 2h, output at <code>~/claude/briefing.md</code>.

=== Anomaly Alerts ===

<code>/home/debian/scripts/claude/anomaly-alerts.sh</code>

Fires Discord alerts for high-score signals. Threshold: <code>min_score=2.0</code>. Excludes: redditwatch, osint-health, socrata:*, openfda:*. JSON validity guard prevents parse errors when anomalywatch is restarting.

== Orchestration ==

=== osint-scan.sh ===

<code>/home/debian/scripts/scripts/osint/osint-scan.sh</code>

Tiered scan intervals via cron:
* 5 min: honeypot, owntracks
* 15 min: skywatch, masintwatch, countywatch, changewatch
* 30 min: donorwatch, egoscan, contractwatch
* 60 min: filingwatch, 990watch, transitwatch
* 240 min: AIS collector (PM2-managed, runs continuously)

Internal routing: <code>curl --max-time N -H "Host: app.tools.ejfox.com" "http://localhost:7777/api/scan?secret=..."</code>

Note: Cloudflare 524s occur if scanning via public hostname -- always use localhost with Host header.

=== health-check.sh ===

<code>/home/debian/scripts/scripts/osint/health-check.sh</code>

Checks recent activity in each app DB using rolling windows (last 4h for skywatch, last 8h for riverwatch) rather than calendar-day counts. This prevents false midnight/8am "nothing today" alerts.

=== Weekly Rollup ===

<code>/home/debian/scripts/weekly-rollup.sh</code>

Sunday 9 AM. Queries anomalywatch <code>/api/intelligence/summary</code>, wiki stats, kanban, entityhub, captures DB. Saves to <code>~/claude/rollups/YYYY-WWW.md</code>.

== Token / Auth Patterns ==

Anomalywatch bearer token loading order (used by all collectors):
# <code>/opt/docker/smallweb/data/_shared/.anomalywatch-token</code>
# Fallback: <code>/opt/docker/smallweb/data/anomalywatch/data/.api-token</code>

Collector scan secret: <code>OSINT_SECRET</code> env var (set in smallweb env or .env file).

== Known Issues / Quirks ==

* <code>localhost</code> in Deno postgres client resolves to <code>::1</code> (IPv6) first; postgres-central binds only to 127.0.0.1. Always use explicit <code>"127.0.0.1"</code> in PG_CONFIG.
* Smallweb app DB files owned by www-data. Running <code>sqlite3</code> as debian creates -shm/-wal files with wrong ownership, breaking app writes. Check ownership after any direct CLI queries.
* Cron environment does not load .bashrc or NVM. Scripts that call <code>claude</code> CLI or use env vars from .bashrc must export PATH and env vars explicitly.
* Cloudflare tunnel adds latency -- anomalywatch HTTPS reports from collectors need at least 30s timeout (not 10s).
* AIS collector anomalywatch reports must use HTTPS (not localhost) because Deno fetch cannot override Host headers for HTTP.

== Related Pages ==

* [[OSINT Monitoring]] -- overview and tool descriptions
* [[Anomalywatch]] -- central signal hub
* [[Skywatch]] -- aviation monitoring
* [[Entityhub]] -- entity knowledge graph
* [[Entity Index]] -- entities with wiki articles
* [[Hudson River AIS Intelligence Brief]] -- vessel intelligence context
* [[Investigations]] -- investigation methodology
* [[Chester ICE Facility]] -- primary investigation subject
* [[Orange County Jail - ICE Detention]] -- active ICE detention in HV
* [[ICE]] -- most-tracked entity (189 mentions)
* [[Akima LLC]] -- key ICE contractor, appears in contractwatch
* [[VPS]] -- server infrastructure
* [[Hudson Valley]] -- geographic focus

[[Category:OSINT]]
[[Category:Infrastructure]]
[[Category:Hudson Valley]]
[[Category:Projects]]