Hop Data Inspector

Right-click a transform and choose Inspect data…​. The inspector runs a separate local pipeline clone. It retains the required upstream transforms and substitutes sinks for downstream transforms outside the inspected section. The original editor pipeline is not changed.

The plugin is installed under plugins/misc/hop-geometry-inspector. Install the Geometry Type plugin separately. Geometry Type owns the shared Geometry/JTS, GeoTools, Imagen, ImageIO-Ext and UOM runtime; do not copy these libraries into the Inspector folder. Apache Hop 2.19.0 and Java 21 are required; Java 25 is included in compatibility tests.

Every captured Hop row is inspectable, including rows without geometry, null geometry, empty geometry and malformed geometry. Geometry detection accepts native Geometry values and geometry-compatible WKT/EWKT or WKB fields. Parsing failures do not remove the original row from the table.

Start opens the separate stream chooser. Initial stream selection determines the initial selection; confirm the streams to capture in this second dialog.

The stream chooser is authoritative. Select more than one stream to capture several results in a single execution. Each stream has its own metadata and row count. MAIN captures regular putRow output; TARGET captures explicitly routed putRowTo output. ERROR captures the configured error output, including error fields. INPUT identifies the incoming peer. Unused outputs can be empty. Optional ports are not activated by inspection.

FIRST keeps the first N rows, LAST the last N, and RANDOM uses reservoir sampling. FIRST stops when all selected streams have reached N, the pipeline ends, the timeout expires or capture is cancelled. LAST and RANDOM run until completion, cancellation or timeout. The default is 1,000 rows. ALL records a full disk cache instead of retaining an unlimited list in the heap.

A completed sample is not a complete stream. Samples are never accepted as replay inputs.

Each captured stream becomes a result layer with its own table. Rows remain read-only. The layer label identifies transform, stream, capture timestamp, sample/cache mode and completion. Further inspections in the same pipeline editor add layers to the existing workspace.

Toolbar icons are represented by text above: Extent means Zoom to extent, Refresh means Refresh geometry view, Emphasize means Emphasize small features, and Background means Toggle background map. Layer labels and the status line are abbreviated. Selecting a row fills the attribute table and geometry text.

Open Data Inspector caches / replay…​ from a transform context menu. If the list is empty, use Record complete cache…​ or open Inspect data…​, select Record complete disk cache, and choose the transform’s OUTPUT MAIN stream. Select caches to open or delete. An open cache is protected from automatic or manual deletion. Caches survive Hop restarts; interrupted recordings remain incomplete and cannot be replayed.

Defaults: ${user.home}/.hop/data-inspector/cache, 10 GiB overall and seven days since last access. Override through JVM properties geometryInspector.cache.path, geometryInspector.cache.bytes and geometryInspector.cache.days. Oldest unused caches are removed first. If protected data fills the budget, capture fails instead of marking truncated data complete.

Rows use Hop’s typed binary serialization, including Geometry Type values. The cache is not a portable exchange format. Unsupported value types and schema changes fail that recording. The manifest stores provenance, schema version, generation, row count, completion and a fingerprint; credentials are not written into the manifest. Captured data can itself contain sensitive fields.

Completeness and configuration freshness are distinct. Only successful full recordings can be used for replay, and their upstream fingerprint must still match. Downstream-only edits do not invalidate upstream data. External databases/files are not polled for freshness: check the timestamp and capture again when source contents may have changed.

Choose caches and Run TO, Run FROM or Run BETWEEN. The context transform is the starting selection; choose the second endpoint for BETWEEN. The execution dialog lists caches and every transform to execute, including writers. Confirming execution runs that section with its normal side effects. A Vector Reader (SOGIS_VECTOR_READER) can execute normally in a replay section; when it is replaced by a complete cache, its external file is not read again.

Multiple inputs must originate from the same capture generation. There is no silent fallback to a fresh full execution when required caches are absent, incomplete, incompatible or stale.

The first implementation accepts a conservative set of standard row-stream transforms plus the Vector Reader. Unsupported plugins are rejected before engine initialization. It excludes cycles, partitioning, multiple copies, non-normal pipeline modes and nested execution. Cache boundaries currently require logical MAIN output; directed/error caches are inspectable but are not accepted as replay sources.

Apache Hop Desktop with the local, normal pipeline engine. This is a GUI plugin, not a transform to insert into a pipeline. Hop Web and remote/distributed engines are not supported.

Download the pipelines with their adjacent README and data files:

See example instructions. Automated fixtures in e2e/ are separate from these user-facing instructions.

Download configured error-stream example.

Capture status distinguishes natural completion, sample limit, timeout, cancellation and pipeline failure. Full caches require natural, error-free completion. Parse failures remain visible as rows. Malformed cache files fail with a row/schema diagnostic; they are never silently repaired into a replayable result. Cache/replay validation errors appear before execution where possible.

No reprojection, geometry editing, raster, point clouds, 3D, automatic capture of normal runs or external-source freshness detection. Replay does not restore arbitrary transform state or external side effects. A sample cannot reveal errors outside the captured rows.

After updating plugin JARs, restart Hop. The development sync script replaces the plugin directory to avoid stale classes. Keep Geometry Type installed alongside Inspector, with its shared runtime. Use the Hop log for pipeline/capture failures and the viewer status for background-map diagnostics. Check a known LV95 feature against swisstopo, then pan/zoom/resize and repeat on a HiDPI display.

CaptureService clones/prunes pipelines and wraps the original Hop row handlers. StreamDescriptor identifies direction, peer, output kind and transform copy. InspectionResult keeps stream provenance, capture generation and completeness separate from its RowStore.

MemoryRowStore owns samples. DiskRowStore stores Hop schema metadata, 8 MiB binary row blocks, per-row CRCs and a disk offset index. CacheRepository manages manifests, reader leases, budgets and cleanup. QueryIndex uses bounded chunks and two-way external merges, with capture ordinal as the stable tie-breaker.

ReplayPlanner validates the graph and cache inputs before preparing an engine. Internal ReplaySourceMeta instances are only placed in the execution clone and are never saved into the user pipeline. The original upstream source is not initialized when replaced by a cache source.

The existing Geometry Type shared classloader remains the owner of Geometry/JTS and the common GeoTools/Imagen/ImageIO-Ext/UOM runtime. Inspector workers use the plugin context classloader, including metadata serialization, geometry parsing and WMTS rendering.

Follow AGENTS.md for the exact Geometry dependency, Maven settings and JDK setup. Run mvn clean verify, python3 scripts/verify-package.py and the installed classloader test against the built Inspector/Geometry ZIPs. Compatibility tests use JDK 25. Linux SWT tests require Xvfb.

Unit/integration tests cover typed rows, disk queries, capture routing and replay source replacement. Installed-plugin tests validate runtime classloader identity and serialization through packaged JARs. User examples use standard Hop transforms; test fixtures and expected data live under e2e/.

The common hop-plugin-parent enables doclint during Maven verify. Its transform rules do not cover GUI plugins; scripts/check-docs.py validates the Inspector handbook and example XML separately. python3 scripts/build-docs-site.py --serve builds the working tree. CI passes --revision for the exact checked-out commit. Site validation checks local links, anchors, downloads and search entries.

GitHub Pages must use the GitHub Actions deployment source. Pull requests only build; main deploys. Existing Maven workflow references, canonical ZIP publication and installed-test gates are preserved.

Run mvn test -DgeometryInspector.swtSmoke=true on a desktop (under xvfb-run -a on Linux). The parent supplies -XstartOnFirstThread on macOS. Tests exercise the real SWT viewer and the background settings dialog; screenshots are written under target/.

For manual acceptance, open examples/geometry/inspect-geometry.hpl, inspect Source, select the null and malformed rows, sort by id and change the geometry field. Details must follow the selected row. Add the Comparison result from the layer example, change visibility/order/opacity and pick overlapping geometry. Try an unavailable background URL and verify that rows remain selectable. Close the pipeline editor and confirm its Inspector workspace closes. Reopen a full cache after restarting Hop and compare the results of the linear and two-input replay examples.

Disk writes apply backpressure on the background capture threads rather than queueing unbounded rows. Query runs contain at most 4096 ordinals; external merges use two input cursors. Each row has an integrity checksum checked during reads and replay preflight.