Migrate to Apache Tika 4.0.0-beta-1 by dadoonet · Pull Request #2432 · dadoonet/fscrawler

dadoonet · 2026-07-07T15:58:04Z

Summary

Migrates FSCrawler from Apache Tika 3.3.1 to Tika 4.0.0-beta-1.

Bump tika.version to 4.0.0-beta-1 and mark tika-parsers-standard-package as <type>pom</type> (it is a POM aggregator in Tika 4).
Rework TikaInstance for the Tika 4 API:
- Parser.parse() now requires a TikaInputStream (wrapped in extractText).
- Tesseract paths moved from the parser to TesseractOCRConfig; the parser is built from that config.
- PDF OCR strategy moved from PDFParser.setOcrStrategy(String) to PDFParserConfig.setOcr(OcrConfig) with an enum (FSCrawler string values are mapped, ocr_and_text stays the default).
- Use TikaCoreProperties.RESOURCE_NAME_KEY instead of the literal resourceName (renamed to X-TIKA:resourceName).
ExternalParser.check() was removed in Tika 4; the OCR IT now skips via @DisabledIf + TesseractOCRParser.hasTesseract().
Transitive dependencies verified: homogeneous Tika 4, no version conflicts, no Jetty/CXF/tika-server pulled in; Tika's Jackson 2 and FSCrawler's Jackson 3 coexist by namespace.

Breaking changes

fs.tika_config_path is temporarily unsupported: Tika 4 removed the XML configuration-file mechanism. FSCrawler now fails fast with a clear message if the setting is used. Support is expected to return on top of Tika 4's JSON configuration.
Several meta.raw.* metadata key names changed (documented in the 2.10 release notes):
- Image/EXIF keys are namespaced under img: (e.g. Number of Tables → img:Number of Tables, Exif IFD0:* → img:Exif IFD0:*).
- resourceName → X-TIKA:resourceName.
- New keys such as Content-Type-Magic-Detected, X-TIKA:Parsed-By-Full-Set and text encoding-detection keys.

Documentation

Refreshed the raw metadata examples (PDF/MP3) and the REST examples to match Tika 4 output.
Documented the tika_config_path removal and the meta.raw.* key changes in the 2.10 release notes.

Test plan

Unit tests: mvn clean test -DskipIntegTests (tika module green).
Full reactor build: mvn clean install -DskipTests -Ddocker.skip (16 modules, enforcer + Spotless pass).
Integration tests (TestContainers + local Tesseract): FsCrawlerTestOcrIT (ocr + ocr_disabled) and FsCrawlerRestIT#documentWithExternalTags green.
Swept the other metadata ITs for img:/hasSize regressions (RawIT and SubDirsIT unaffected).
Full Docker build and OCR image build (not run here).

Known upstream issue

Non-ASCII HTML metadata attribute names are mis-decoded by Tika 4 (tracked upstream in TIKA-4774); the affected unit assertion is annotated accordingly.

Follow-up

Re-introduce custom Tika configuration on top of Tika 4's JSON configuration mechanism.

🤖 Generated with Claude Code

Note

High Risk
Core document parsing and indexed field names change across the ingestion path; custom Tika XML config is rejected and PDF OCR behavior differs unless ocr_and_text is set explicitly.

Overview
Upgrades FSCrawler from Tika 3.3.1 to 4.0.0-beta-1, including Maven tika-parsers-standard-package as a POM aggregator.

TikaInstance is adapted to Tika 4: parsing goes through TikaInputStream with TemporaryResources; Tesseract paths and PDF OCR strategy use TesseractOCRConfig and PDFParserConfig/OcrConfig instead of the old parser APIs. fs.tika_config_path now fails fast with FsCrawlerIllegalConfigurationException because XML Tika config was removed upstream.

Defaults and breaking metadata: fs.ocr.pdf_strategy default changes from ocr_and_text to auto. Indexed meta.raw.* keys shift (e.g. resourceName → X-TIKA:resourceName, image/EXIF under img:, plus new keys like Content-Type-Magic-Detected). Docs, release notes, and unit/integration tests are updated to match.

OCR integration tests use TesseractOCRParser.hasTesseract() via @DisabledIf instead of removed ExternalParser.check().

^{Reviewed by Cursor Bugbot for commit 281d916. Bugbot is set up for automated code reviews on this repo. Configure here.}

mergify · 2026-07-07T15:58:53Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7034a29. Configure here.}

- Switch metadata listing to .sorted(String.CASE_INSENSITIVE_ORDER) in TikaDocParser - Reorder the raw-metadata assertions in TikaDocParserTest to match the new order - Drop a redundant duplicate xmpDM:genre entry in the mp3 assertions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… HTML metadata attribute names

…Tesseract - Tika 4 removed the static ExternalParser.check(); replace the Tesseract availability probe with the same hasTesseract() check used in production - Convert the runtime Assumptions skip into a declarative @DisabledIf so the test reports as disabled (with a reason) instead of aborted - Drop the now-unused Assumptions import Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ika 4 - Refresh PDF (47 keys) and MP3 (20 keys) raw metadata examples in local-fs to match Tika 4 output (X-TIKA:*, Content-Type-Magic-Detected, pdf:docinfo:*, access_permission:*), sorted case-insensitively like the real output - Fix the REST examples in rest.rst (X-TIKA:Parsed-By and friends) - Warn that fs.tika_config_path is temporarily unsupported on Tika 4 (XML configuration mechanism removed upstream); note it in the 2.10 changelog Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…mespace - Tika 4 namespaces all metadata-extractor image keys under "img:" (both the bare keys like "Number of Tables" and the "Exif IFD0:/SubIFD:" ones); values are unchanged, ICC:/tiff:/Content-*/X-TIKA: keys stay as-is - Bump hasSize(64) -> hasSize(66) (Tika 4 adds Content-Type-Magic-Detected and Content-Length) and add the img: prefixes, verified against real output - Sort the raw-metadata assertions case-insensitively for readability Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Tika 4 renamed the resourceName metadata key to X-TIKA:resourceName; update the JsonPath in documentWithExternalTags to bracket notation for the new key Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Note the img: namespace for image/EXIF metadata keys - Note resourceName -> X-TIKA:resourceName rename - Note new keys: Content-Type-Magic-Detected, X-TIKA:Parsed-By-Full-Set and the text encoding-detection keys (X-TIKA:detectedEncoding/encodingDetectionTrace/encodingDetector) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TikaInputStream.get(stream, metadata) closes the underlying stream on exit, which breaks FTP uploads where completePendingCommand() must run in the caller's close path. Use TemporaryResources so Tika can spool without closing provider streams. Co-authored-by: Cursor <cursoragent@cursor.com>

…mance

sonarqubecloud · 2026-07-13T08:17:48Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

dadoonet force-pushed the tika-4.x branch from cf70201 to 7034a29 Compare July 10, 2026 17:37

cursor Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread tika/src/main/java/fr/pilato/elasticsearch/crawler/fs/tika/TikaInstance.java

dadoonet and others added 14 commits July 13, 2026 10:16

Update to Tika 4

8d5f1ed

fix(tika): 🐛 clarify removal of XML configuration support in Tika 4

1e441f2

fix(test): 🐛 update TikaDocParserTest to handle mis-decoded non-ASCII…

bcbd691

… HTML metadata attribute names

fix(log): 🐛 change fscrawler.metadata logger level from debug to info

221549f

Update to Tika 4

61ab6d2

fix(log): 🐛 change fscrawler.fs.client logger level from debug to info

0717288

fix(test): 🐛 read X-TIKA:resourceName in REST IT for Tika 4

897c84e

- Tika 4 renamed the resourceName metadata key to X-TIKA:resourceName; update the JsonPath in documentWithExternalTags to bracket notation for the new key Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ocr): 🐛 change default PDF strategy to 'auto' for improved perfor…

281d916

…mance

dadoonet force-pushed the tika-4.x branch from 50008b4 to 281d916 Compare July 13, 2026 08:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate to Apache Tika 4.0.0-beta-1#2432

Migrate to Apache Tika 4.0.0-beta-1#2432
dadoonet wants to merge 14 commits into
masterfrom
tika-4.x

dadoonet commented Jul 7, 2026 •

edited by cursor Bot

Loading

Uh oh!

mergify Bot commented Jul 7, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

dadoonet commented Jul 7, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking changes

Documentation

Test plan

Known upstream issue

Follow-up

Uh oh!

mergify Bot commented Jul 7, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jul 13, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dadoonet commented Jul 7, 2026 •

edited by cursor Bot

Loading