coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6b0334affebca4f9f5b28ed8dcb0831814c1957a
rose-ash/spec/tests/hyperscript-upstream-manifest.md
giles fd1dfea9b3 HS tests: scrape v0.9.90 upstream in full, flip silent stubs to loud SKIPs 
- scrape-hs-upstream.py: new scraper walks /tmp/hs-upstream/test/**/*.js
  and emits body-style records for all 1,496 v0.9.90 tests (up from 831).
  Widens coverage into 66 previously-missing categories — templates,
  reactivity, behavior, worker, classRef, make, throw, htmx, tailwind,
  viewTransition, and more.

- build-hs-manifest.py + hyperscript-upstream-manifest.{json,md}:
  coverage manifest tagging each upstream test with a status
  (runnable / skip-listed / untranslated / missing) and block reason.

- generate-sx-tests.py: emit (error "SKIP (...)") instead of silent
  (hs-cleanup!) no-op for both skip-listed tests and generator-
  untranslatable bodies. Stub counter now reports both buckets.

- hyperscript-feature-audit-0.9.90.md: gap audit against the 0.9.90
  spec; pre-0.9.90.json backs up prior 831-test snapshot.

New honest baseline (ocaml runner, test-hyperscript-behavioral):
  831 -> 1,496 tests; 645 -> 1,013 passing (67.7% conformance).
  483 failures split: 45 skip-list, 151 untranslated, 287 real.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 20:27:22 +00:00

164 lines
7.7 KiB
Markdown
Raw Blame History

# _hyperscript v0.9.90 upstream test coverage manifest
- Upstream tag: `v0.9.90` (commit `a13de2ca`, 2026-04-13)
- Our snapshot JSON: `spec/tests/hyperscript-upstream-tests.json` (831 tests, scraped 2026-04-09)
- Total upstream tests: **1496**
- Runnable (present + not skip-listed): **764** (51.1%)
- Skip-listed (present but guarded in generator): **44** (2.9%)
- Missing from our snapshot: **688** (46.0%)
Current conformance: runnable / total = **764/1496 = 51.1%**.
## Per-category
| Category | Upstream | Runnable | Skip-listed | Missing |
|---|---:|---:|---:|---:|
| add | 19 | 19 | 0 | 0 |
| api | 1 | 0 | 0 | 1 |
| append | 13 | 13 | 0 | 0 |
| arrayIndex | 14 | 0 | 0 | 14 |
| arrayLiteral | 8 | 0 | 0 | 8 |
| asExpression | 42 | 17 | 0 | 25 |
| askAnswer | 5 | 5 | 0 | 0 |
| assignableElements | 8 | 8 | 0 | 0 |
| asyncError | 2 | 2 | 0 | 0 |
| attributeRef | 22 | 1 | 0 | 21 |
| beep! | 6 | 0 | 0 | 6 |
| behavior | 10 | 0 | 0 | 10 |
| bind | 44 | 38 | 0 | 6 |
| blockLiteral | 4 | 0 | 0 | 4 |
| boolean | 2 | 0 | 0 | 2 |
| bootstrap | 26 | 14 | 0 | 12 |
| breakpoint | 2 | 0 | 0 | 2 |
| call | 6 | 6 | 0 | 0 |
| classRef | 9 | 0 | 0 | 9 |
| closest | 10 | 3 | 0 | 7 |
| collectionExpressions | 28 | 22 | 0 | 6 |
| comparisonOperator | 83 | 40 | 0 | 43 |
| component | 20 | 18 | 0 | 2 |
| cookies | 5 | 1 | 0 | 4 |
| def | 27 | 24 | 3 | 0 |
| default | 15 | 9 | 0 | 6 |
| dialog | 12 | 9 | 0 | 3 |
| dom-scope | 25 | 25 | 0 | 0 |
| empty | 13 | 13 | 0 | 0 |
| evalStatically | 8 | 8 | 0 | 0 |
| eventsource | 13 | 0 | 0 | 13 |
| fetch | 23 | 15 | 8 | 0 |
| focus | 3 | 3 | 0 | 0 |
| functionCalls | 12 | 0 | 0 | 12 |
| go | 5 | 5 | 0 | 0 |
| halt | 7 | 7 | 0 | 0 |
| hide | 16 | 14 | 0 | 2 |
| hs-include | 10 | 0 | 0 | 10 |
| htmx | 9 | 0 | 0 | 9 |
| idRef | 4 | 0 | 0 | 4 |
| if | 19 | 19 | 0 | 0 |
| in | 10 | 1 | 0 | 9 |
| increment | 20 | 20 | 0 | 0 |
| init | 3 | 3 | 0 | 0 |
| js | 11 | 1 | 0 | 10 |
| live | 23 | 23 | 0 | 0 |
| liveTemplate | 16 | 10 | 0 | 6 |
| log | 4 | 4 | 0 | 0 |
| logicalOperator | 10 | 3 | 0 | 7 |
| make | 8 | 0 | 0 | 8 |
| mathOperator | 15 | 5 | 0 | 10 |
| measure | 6 | 2 | 0 | 4 |
| morph | 10 | 10 | 0 | 0 |
| no | 9 | 5 | 0 | 4 |
| not | 9 | 0 | 0 | 9 |
| null | 1 | 0 | 0 | 1 |
| numbers | 1 | 0 | 0 | 1 |
| objectLiteral | 12 | 1 | 0 | 11 |
| on | 70 | 27 | 33 | 10 |
| parser | 14 | 7 | 0 | 7 |
| pick | 24 | 7 | 0 | 17 |
| positionalExpression | 7 | 0 | 0 | 7 |
| possessiveExpression | 23 | 0 | 0 | 23 |
| propertyAccess | 12 | 0 | 0 | 12 |
| pseudoCommand | 11 | 0 | 0 | 11 |
| put | 38 | 38 | 0 | 0 |
| queryRef | 13 | 1 | 0 | 12 |
| reactive-properties | 4 | 4 | 0 | 0 |
| reactivity | 8 | 0 | 0 | 8 |
| regressions | 16 | 0 | 0 | 16 |
| relativePositionalExpression | 23 | 4 | 0 | 19 |
| remove | 19 | 14 | 0 | 5 |
| repeat | 30 | 29 | 0 | 1 |
| reset | 8 | 8 | 0 | 0 |
| resize | 3 | 3 | 0 | 0 |
| runtime | 7 | 0 | 0 | 7 |
| runtimeErrors | 18 | 0 | 0 | 18 |
| scoping | 20 | 1 | 0 | 19 |
| scroll | 8 | 8 | 0 | 0 |
| security | 1 | 0 | 0 | 1 |
| select | 4 | 4 | 0 | 0 |
| send | 8 | 8 | 0 | 0 |
| set | 31 | 25 | 0 | 6 |
| settle | 3 | 1 | 0 | 2 |
| show | 18 | 2 | 0 | 16 |
| socket | 16 | 4 | 0 | 12 |
| some | 6 | 0 | 0 | 6 |
| sourceInfo | 4 | 0 | 0 | 4 |
| splitJoin | 7 | 7 | 0 | 0 |
| stringPostfix | 3 | 0 | 0 | 3 |
| strings | 8 | 0 | 0 | 8 |
| styleRef | 6 | 0 | 0 | 6 |
| swap | 4 | 4 | 0 | 0 |
| symbol | 2 | 0 | 0 | 2 |
| tailwind | 12 | 0 | 0 | 12 |
| take | 15 | 12 | 0 | 3 |
| tell | 10 | 10 | 0 | 0 |
| templates | 48 | 0 | 0 | 48 |
| throw | 7 | 0 | 0 | 7 |
| toggle | 25 | 25 | 0 | 0 |
| tokenizer | 17 | 0 | 0 | 17 |
| transition | 17 | 17 | 0 | 0 |
| trigger | 6 | 0 | 0 | 6 |
| typecheck | 5 | 0 | 0 | 5 |
| unlessModifier | 1 | 0 | 0 | 1 |
| viewTransition | 9 | 0 | 0 | 9 |
| wait | 7 | 7 | 0 | 0 |
| when | 41 | 41 | 0 | 0 |
| worker | 1 | 0 | 0 | 1 |
| **TOTAL** | **1496** | **764** | **44** | **688** |
## What unlocks how many — MISSING tests by block_reason
| Block reason | Missing | Example | Est. effort |
|---|---:|---|---|
| `unscraped-category` | 282 | breakpoint / parses as a top-level command | low — extend scraper to cover these upstream files |
| `unscraped-in-known-category` | 170 | default / can default variables | low — re-scrape; file was walked but these cases missed |
| `added-post-snapshot` | 122 | hide / can hide via the hidden attribute strategy | low — re-scrape upstream, bump JSON snapshot |
| `needs-pattern:${}` | 51 | liveTemplate / script type="text/hyperscript-template" works as a l... | low — template string interpolation in HS parser |
| `needs-script-tag` | 26 | throw / can throw a basic exception | medium — emit <script type="text/hyperscript"> wrapper in generator |
| `needs-websocket` | 12 | socket / with timeout parses and uses the configured timeout | high — WebSocket mock server |
| `needs-css-transitions` | 8 | viewTransition / runs the body when view transitions API is unavail... | medium — transitionend event dispatch in fixtures |
| `needs-pattern:<sel/>` | 8 | comparisonOperator / exists works | low — parser rule for <sel/> positional expr |
| `needs-pattern:[@attr]` | 5 | attributeRef / attributeRef with no value works | low — attribute-ref parser rule |
| `needs-dialog-api` | 3 | dialog / show opens a dialog (non-modal) | low — stub showModal/close on HTMLDialogElement in fixture DOM |
| `needs-intersection-observer` | 1 | on / on intersection fires when the element is in the viewport | medium — IntersectionObserver mock |
## Skip-listed tests by block_reason
| Block reason | Skipped | Example | Est. effort |
|---|---:|---|---|
| `translation-TBD` | 39 | fetch / can do a simple fetch w/ html | unknown — needs case-by-case generator work |
| `needs-script-tag` | 5 | def / functions can be namespaced | medium — emit <script type="text/hyperscript"> wrapper in generator |
## How this was built
1. `git clone --depth 1 --branch v0.9.90 https://github.com/bigskysoftware/_hyperscript /tmp/hs-upstream`; `git fetch --unshallow` for dated history.
2. Walked `test/` (excluding `vendor/`, `manual/`, `fixtures.js`, `global-*.js`, `entry.js`, `htmx-fixtures.js`, `playwright.config.js`).
3. For each `.js` file, a small Python parser finds every `test.describe(...)` block, then every `test(...)` within it — balanced-paren scan that ignores strings, regex literals, line/block comments.
4. Category = filename stem (e.g. `add.js` → `add`). Test name = the first string literal argument of `test(...)`.
5. Matched each upstream test against `spec/tests/hyperscript-upstream-tests.json` using `(category, name-normalized)` keys (whitespace-collapsed lowercase). Copied `complexity` when found; inferred it otherwise from body content (sinon./script-tag/dialog/Promise/evaluate).
6. `status = runnable` if present and name not in generator's `SKIP_TEST_NAMES`; `skip-listed` if present and in that set; `missing` otherwise.
7. `block_reason` classified from body content — sinon./script tag/dialog/worker/eventsource/WebSocket/MutationObserver/transition/focus/ResizeObserver patterns or `<sel/>`/`${}`/`[@attr]` HS syntax. Missing tests in files touched between 2026-04-09 and 2026-04-14 (`git log --after --before -- test/`) are tagged `added-post-snapshot`.
8. Regenerate: `python3 tests/playwright/build-hs-manifest.py` (expects `/tmp/hs-upstream` clone at `v0.9.90`).
## Untranslated caveat
The markdown reports `runnable = present + not skip-listed`. Empirical baseline is 645 pass / 109 fail / 77 skip on 831 present tests. The ~109 failures are in-scope but reveal implementation gaps; they are not statically identifiable from upstream source without running the generator. This manifest therefore does not surface an `untranslated` bucket — treat the 109 empirical fails as the lower bound of that bucket inside the `runnable` count.
Reference in New Issue View Git Blame Copy Permalink