virgiliojr94/book-to-skill

What happened The BeautifulSoup text conversion inserts a newline between every text node, including nodes separated only by inline styling. A span inside a heading, word, or syntax-highlighted code block changes the extracted content. This affects two current paths with the same cause: HTML: the bs4 fallback when Trafilatura is unavailable or does not return usable content. EPUB: the preferred ebooklib + bs4 parser. This path does not use Trafilatura. On unmodified master, a synthetic two-chapter source using Chapter <span>1</span> produces 0 detected chapters. Removing only the span markup produces 2. The styled source still exits successfully through the full CLI, including HTML --mode technical. For code, this source: <pre><code>if enabled: <span>print</span>("ok") </code></pre> becomes: if enabled: print ("ok") Compiling that extracted snippet raises IndentationError; the same snippet without the styling spans, and the current stdlib output for this fixture, compile su

What happened reuse_is_safe() returns (True, "workdir intact and all sources match the current inputs") when the extraction corpus has been deleted, emptied, or replaced with unrelated content. The original input file and its recorded SHA-256 are unchanged in all three cases. Step 0 names this helper as the executable form of the interrupted-run recovery check. Its result can therefore authorize reuse of a missing or corrupted full_text.txt, even though later generation steps read that file. This is a follow-up to the merged source-fingerprint protection in #224: the original-source check works, but it does not establish the integrity of the extraction output. I am not claiming that an actual host agent resumed and generated a wrong skill in this reproduction; the reproduced failure is the helper's incorrect safety verdict. What you expected Unchanged inputs are necessary for reuse, but the extracted corpus must also exist, be readable, and match the completed extraction. Missing or ch

Problem book_to_skill/sanitize.py::is_invisible_codepoint() (sanitize.py:121) is an enumerated blocklist organised by attack shape into eight commented groups. Group 1 already strips U+180E MONGOLIAN VOWEL SEPARATOR (sanitize.py:17) — but the free variation selectors in the same Mongolian block, and two further Default_Ignorable channels, are not in the list. They pass through extraction untouched, and because tools/scan_generated_skill.py imports the same predicate (scan_generated_skill.py:25, consumed by _is_invisible at 113 and the unicode.invisible finding at 241-251), the scanner does not flag them either. That is the exact failure the module documents and exists to prevent (sanitize.py:124-126): Exposed so the generated-skill scanner can flag exactly what extraction strips. When the two sets drift, the extractor lets a character through that the scanner then warns about — or worse, neither layer covers it. The gap, measured is_invisible_codepoint() returns False for nine as

Problem book_to_skill/utils.py::_structural_chapter_count() decides a chapter count by returning len(levels[depth]) for the shallowest heading depth that has at least two distinct titles (utils.py:509). It has no concept of front matter or back matter, so every framing heading at that depth is counted as a teaching chapter. Measured on upstream/master (526f362, v1.4.0) with a synthetic fixture (no copyrighted text) shaped like a real book — 20 titled chapters plus the front matter and back matter a book actually carries: headings role ## Preface, ## Acknowledgements, ## Table of Contents front matter ## Chapter Title 1 … ## Chapter Title 20 the 20 real chapters ## Appendix A: Notation, ## Appendix B: Datasets, ## Glossary, ## Bibliography, ## Index back matter $ python3 scripts/extract.py synthetic-book.md --mode text --install-missing no chapters: 28 (structural) Extraction complete: Chapters: 28 detected overall (structural) ToC : yes metadata.json fro

Problem python3 -m pytest tests/ -q in CI always reports 5 skipped, because the only thing CI installs is pytest (ci.yml:28) and ruff (ci.yml:41). Those 5 tests are gated by pytest.importorskip: Site Dep tests/test_html_block_boundaries.py:131 bs4 tests/test_html_boilerplate_extraction.py:32, 38, 47, 56 trafilatura The consequence is that production fallback behaviour is asserted nowhere in CI — the very behaviour those tests exist to pin. Measured (Ubuntu 20.04, Python 3.9.5, same tree, same interpreter otherwise) pytest tests/ -q -> 698 passed, 5 skipped (ci.yml today: pytest only) pytest tests/ -q (bs4 + trafilatura) -> 703 passed, 0 skipped The skip is purely dependency-gated, proven by flipping one dep at a time: bs4 present, trafilatura absent -> TestConvergenceWithBeautifulSoup 1 passed test_html_boilerplate_extraction 1 passed, 4 skipped trafilatura present, bs4 absent -> test_html_boilerplate_ex