Devseis Endpoint Auditor — stage 1, steps 1–3

A read-only audit of a Windows, Linux or macOS endpoint against a security baseline, GDPR technical controls, EU AI Act controls and organisational controls. It produces:

• a text report in the Devseis format (one line per check, then its status), and
• a JSON evidence file in one shared format, the input for the Devseis auditor model (later steps).

This repository holds steps 1–3 of the MVP: the collectors, the evidence format, the check catalog, a synthetic evidence generator, the control library and the training data builder, so the model can be built before real customer data exists. The generated training data is in the dataset Devseis/endpoint-auditor-synthetic.

Not a certification. ISO 27001 certification needs an accredited auditor, and much of GDPR and AI Act compliance is organisational. The tool collects evidence and flags gaps.

What it checks

Every check, with its ISO 27001:2022 Annex A, GDPR and AI Act references, is in catalog/checks.json. Control titles are referenced; the ISO standard text is not reproduced (it is copyrighted).

Statuses: Compliant, Non-Compliant, NotApplicable, Pending (needs the model or an organisational answer) and Error (value not readable, usually because the collector was not run as administrator/root).

Repository layout

baseline.conf                     required values (password length, lockout, USB, approved admins, AI tools…)
organisational.conf               answers to the organisational questions (yes / no / partial / n/a)
catalog/checks.json               every check: id, section, name per OS, rule, control references
schema/evidence.schema.json       JSON Schema of the evidence file
collectors/windows/audit.ps1      Windows collector (Windows PowerShell 5.1 or PowerShell 7)
collectors/linux/audit.sh         Linux collector (Ubuntu/Debian, RHEL family)
collectors/macos/audit.sh         macOS collector (bash 3.2 as shipped with macOS)
synthetic/generate_evidence.py    synthetic evidence in the same format, using the same rules
synthetic/samples/                three example synthetic records (one per OS)
tools/validate_evidence.py        checks evidence against the schema rules and the catalog
tools/render_report.py            turns evidence back into the text report
library/controls.json             ISO 27001 controls, GDPR and AI Act articles in Devseis's own words
library/check_guidance.json       per check: why it matters, risk, fix steps per OS, how to verify
library/ai_tools.json             AI tools the inventory can find: data location, default AI Act class, GDPR notes
training/build_dataset.py         evidence -> chat-format training examples (finding, summary, AI classification)
training/check_faithfulness.py    checks that every answer is grounded in its prompt

Running an audit

The collectors only read settings. The only files they write are the report and evidence (plus short-lived exports from secedit/auditpol in TEMP on Windows, deleted straight away).

# Windows (elevated PowerShell). Output: C:\ProgramData\Devseis\EndpointAudit\
powershell -ExecutionPolicy Bypass -File .\collectors\windows\audit.ps1

# Linux. Output: /var/log/devseis-endpoint-audit/
sudo ./collectors/linux/audit.sh

# macOS. Output: /Library/Logs/Devseis/EndpointAudit/
sudo ./collectors/macos/audit.sh

Options: --baseline FILE, --org FILE, --output DIR (Windows: -Baseline, -Organisational, -Output). Exit code: 0 all compliant, 1 at least one Non-Compliant, 2 could not run or write output.

Without administrator/root rights the audit still runs; checks that need those rights report Error.

Configuring the baseline

Edit baseline.conf. Important values to set per organisation:

• APPROVED_ADMINS — the IT administrator accounts allowed to be administrators.
• APPROVED_AI_TOOLS — AI tools allowed on endpoints, as named in the AI inventory.
• APPROVED_REMOTE_SERVICES — remote access allowed (e.g. ssh, rdp).
• USB_READ / USB_WRITE — allow or deny.

Answer the eight questions in organisational.conf once; they appear in every report.

Synthetic data

python3 synthetic/generate_evidence.py --count 600 --seed 7 --out synthetic/out/evidence.jsonl
python3 tools/validate_evidence.py synthetic/out/evidence.jsonl
python3 tools/render_report.py synthetic/out/evidence.jsonl --index 0

Records follow the collectors' check order, names and pass/fail rules, across three device profiles (hardened, typical, neglected), current and unsupported OS versions, runs without admin rights, third-party antivirus, and AI tool inventories. Host names, serial numbers and accounts are invented. The same seed always gives the same data.

Control library and training data (steps 2–3)

The library is what the model retrieves at run time; the training examples teach it to turn evidence plus library entries into findings. Three tasks are built from every audit record:

python3 synthetic/generate_evidence.py --count 1000 --seed 7 --out synthetic/out/evidence.jsonl
python3 training/build_dataset.py --evidence synthetic/out/evidence.jsonl --out training/out
python3 training/check_faithfulness.py training/out/*.jsonl

Labels come from templates, not from a language model, so they never contain invented values. Splits are made per audit (80/10/10), so no device appears in two splits. With 1,000 synthetic audits this gives 29,000 examples (23,083 train / 3,123 validation / 2,794 test); all pass the faithfulness check.

Template labels teach the format and grounding. Before customer use, add reviewed real audits and have an ISO 27001 / GDPR professional review the library and a sample of the targets.

Desktop app (Electron + WebLLM)

app/ is the client application. It runs everything on the client's computer:

1. Prepare — downloads the auditor model once through WebLLM (WebGPU) and caches it for offline use. Without WebGPU, the built-in report writer is used instead.
2. Approve — lists every check that will run for this OS; the client clicks Run audit and the OS asks for administrator rights (macOS admin prompt, Windows UAC, Linux pkexec).
3. Progress — each check is shown as it finishes (collectors write --progress lines), then the findings are written.
4. Save — a popup asks where to save the PDF report; with no answer in 60 seconds it is saved to Documents/Devseis Endpoint Audit/<computer>-<date>/ together with the evidence JSON.

Model answers are only used if they pass the same grounding check as the training data; otherwise the template answer is used, so a report is always produced. Scores and counts always come from the evidence.

cd app && npm install && npm test   # prompt parity with training/build_dataset.py
npm start                           # run the app

Offline fine-tuning on CPU: training/train_lora.py (LoRA on Qwen2.5-0.5B-Instruct; see --benchmark and --resume).

Security assistant (chat)

The Space's front page is a chat assistant in the style of Devseis Nox: ask about securing a computer or phone, ISO 27001, GDPR or the EU AI Act, and run an audit from the chat. The model runs on the device with WebLLM (1.5B, about 1 GB on first use); answers are grounded in the Devseis knowledge base (library/knowledge.json, 321 passages generated by library/build_knowledge.py from the catalog and library), cite their sources, and are checked (chatProblems in app/renderer/knowledge.js). Without WebGPU, or if the GPU fails, it answers directly from the knowledge base. Run audit: on phones it starts the phone check (phone.html), which hands the report back to the chat; on computers it leads to the desktop app, or the app's …-evidence.json report can be attached so the chat answers from the user's own results and writes the full report. Conversations stay in the browser. Until model v0.4 (Qwen2.5-1.5B fine-tuned for chat and reports) is published, the chat uses the base Qwen2.5-1.5B-Instruct (chat.version in app/model-version.json).

Install on a Mac

Download from https://devseis-arx.static.hf.space/download.html (universal app for Apple silicon and Intel, macOS 13+; files and checksums in Devseis/endpoint-auditor-releases). Drag it to Applications; on first open, allow it once in System Settings › Privacy & Security › Open Anyway (the build is ad-hoc signed, not yet notarised). The app downloads the AI auditor model on first run and follows the model version in app/model-version.json on this Space. Settings for organisations: ~/Library/Application Support/Devseis Endpoint Auditor/ (baseline.conf, organisational.conf). Build it yourself: cd app && npm install && npm run dist; smoke-test a build with DEVSEIS_SMOKE=<folder> "<app>/Contents/MacOS/Devseis Endpoint Auditor" (add DEVSEIS_SMOKE_MODEL=1 to include the model).

Role of the language model

Code collects the evidence and decides every verdict; the fine-tuned model explains, prioritises and writes the report, and every answer is checked against the evidence before it is used. See docs/MODEL_LLM.md for who does what and how the model's role can grow.

Phone check (iPhone, iPad, Android)

The phone check (phone.html, also reachable from the chat's Run audit) runs entirely in the browser:

1. Detects what a browser can see: OS and version (Safari 26+ hides the iOS version in its user agent, so Safari's own version is used; Android uses client hints), device model (Android), browser, whether a device unlock method is set up (WebAuthn), and WebGPU support.
2. Asks the rest, one question at a time with where to look in Settings (catalog/mobile_questions.json). Every result is labelled detected or self-reported in the evidence and the report.
3. Matches the iOS version against the signed phone subset of the vulnerability bundle (55 KB, signature checked with WebCrypto).
4. Writes the findings with the same auditor model in WebLLM (about 350 MB download, roughly 1 GB of GPU memory, so newer phones) or with the built-in writer, and builds the same report as the desktop app.
5. Saves or shares the report as a PDF (share sheet or download). Phones do not allow a silent auto-save.

mobile/mobile-core.js turns detections and answers into evidence; the synthetic generator calls the same code through mobile/build-evidence.mjs, so training data has exactly the page's wording. Organisation answers can be passed in the link: index.html#org=<base64 JSON {"ORG_ROPA": "yes", ...}>.

Test without a phone: serve the repo (python3 -m http.server 8765) and run app/node_modules/.bin/electron mobile/test/e2e.cjs ios out.json (or android); it answers every question in a phone-sized window with a phone user agent and builds the PDF; then check the evidence with python3 tools/validate_evidence.py out.json.

Vulnerability matching (offline)

vuln.known_vulnerabilities is decided by exact version comparison, never by the model:

1. The collector saves a software inventory next to the evidence (*-inventory.json: names and versions only; it stays on the computer).
2. The app matches it against a signed vulnerability bundle (app/vulndb.cjs): Linux packages against OSV advisories for Ubuntu, Debian, RHEL, AlmaLinux and Rocky (with dpkg/rpm version rules), Windows and macOS applications against NVD version ranges for 33 common business apps (vulndb/products.json), with CISA KEV ("actively exploited") and EPSS scores for priority.
3. Only vulnerabilities with an available fix are counted. Distributions or apps outside the bundle are reported as not covered, never as "0 vulnerabilities". Data older than 14 days is flagged in the finding.

Devseis builds the bundle with python3 vulndb/build_bundle.py (needs internet; about 1 GB of source downloads, 5 MB result) and signs it with node vulndb/sign-bundle.mjs (private key in ~/.devseis/, never in the repository). Test a run without the app: node tools/match_vulns.cjs <evidence.json>. Packaging must ship vulndb/out/ inside the app's audit resources.

Version status

Test status (collector version 0.1.0)

Privacy

Real evidence contains the host name, serial number, user account names and file paths: it is personal data under GDPR. Keep it on the device or in the organisation's own storage. Only synthetic data belongs in this repository.

Roadmap

1. Collectors, evidence format, catalog, synthetic data ← done
2. Control library in Devseis's own wording, for retrieval by the model ← done (expert review pending)
3. Training data: evidence → auditor findings (explanation, risk, recommendation, references) ← done on synthetic data; expert review of a sample pending
4. Fine-tune a small open-weight model (LoRA/QLoRA) to run locally
5. Evaluate on held-out data (correct references, no invented values) and publish the scores
6. Package: one installer per OS that detects the OS, runs the collector and writes the report locally

Licence and citation

Open source by Devseis:

Keep the NOTICE file when redistributing, and credit "Devseis Endpoint Auditor by Devseis". To cite it, use CITATION.cff:

@software{devseis_endpoint_auditor_2026,
  author  = {{Devseis}},
  title   = {Devseis Endpoint Auditor: a local, read-only endpoint auditor for ISO 27001, GDPR and the EU AI Act},
  year    = {2026},
  version = {0.2.0},
  url     = {https://huggingface.co/spaces/Devseis/Arx},
  license = {Apache-2.0}
}

Contributions are welcome: open a discussion on the Space. Reviews of the control library and fix guidance by security practitioners are especially useful.