Jobsub ID 303508.0@dunegpschedd01.fnal.gov

Jobsub ID	303508.0@dunegpschedd01.fnal.gov
Workflow ID	12686
Stage ID	4
User name	usher@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2026-02-03 16:36:07
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2026-02-03 16:42:12
From worker node	Hostname	wn-sate-022.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-02-03 16:37:14
Input files	usertests:prod_monte-carlo-012686-000009_gen_g4_detsim.root
Jobscript	Exit code	0
	Real time	2m (158s)
	CPU time	1m (119s = 75%)
	Max RSS bytes	1033486336 (985 MiB)
Outputting started	2026-02-03 16:39:52
Output files	usertests:prod_monte-carlo-012686-000009_gen_g4_detsim_reco.root usertests:prod_monte-carlo-012686-000009_gen_g4_detsim_reco_larcv.root
Finished	2026-02-03 16:42:12
Saved logs	justin-logs:303508.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

due to invalid 2d index 0
 ... due to invalid 3d index 11137

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 48th record. run: 210009 subRun: 0 event: 48 at 03-Feb-2026 17:39:49 CET
++++>>>> total num hits: 120, num free: 120

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 574!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 16626
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 6913

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 49th record. run: 210009 subRun: 0 event: 49 at 03-Feb-2026 17:39:49 CET
++++>>>> total num hits: 25, num free: 25

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 38!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 927
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 5

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 50th record. run: 210009 subRun: 0 event: 50 at 03-Feb-2026 17:39:50 CET
++++>>>> total num hits: 206, num free: 206

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 577!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 5595
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 475

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
03-Feb-2026 17:39:51 CET  Closed output file "prod_monte-carlo-012686-000009_gen_g4_detsim_reco.root"
03-Feb-2026 17:39:51 CET  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/63/7c/prod_monte-carlo-012686-000009_gen_g4_detsim.root"
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_mpv_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing neutrino_mpv_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_mcst_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_masked_true2reco_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_index_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_masked_true_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_highE_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_lowE_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_leftover_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_pcluster_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_at_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_chi2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_occupancy_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_nhits_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_cryo_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tpc_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge0_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge1_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time0_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time1_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key0_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key1_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_sed_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_ghost_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_corrected_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_rescaled_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Closing output file

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.272438       1.72337       8.62173       1.21547       1.49494        50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000306446   0.00188447      0.01021     0.000589999   0.00317034       50     
reco:gaushit:GausHitFinder                     0.040357      0.0810373     0.210507      0.0673585     0.0400061       50     
reco:cluster3d:Cluster3D                      0.00111139     0.154291       2.72605      0.0258825     0.465057        50     
reco:opdec:Deconvolution                       0.0405125     0.171914      0.314979      0.166115      0.0642214       50     
reco:ophitspe:OpHitFinderDeco                  0.0275691     0.0319867     0.0425537     0.0314984    0.00299404       50     
reco:opflash:OpFlashFinder                    0.000240452   0.00111623    0.00324453    0.00109336    0.000547873      50     
reco:opslicer:OpSlicer                        0.00029755      0.18224      0.628859      0.137041       0.16564        50     
reco:mcreco:MCReco                             0.0287835     0.197659      0.820782      0.101369      0.200172        50     
[art]:TriggerResults:TriggerResultInserter    1.4727e-05    2.56744e-05   8.6262e-05    2.0849e-05    1.44723e-05      50     
end_path:superaMC:LArSoftSuperaDriver          0.104691      0.795053       3.88483       0.51927      0.726893        50     
end_path:out1:RootOutput                       4.789e-06    8.80796e-06   1.9757e-05     8.165e-06    3.26572e-06      50     
end_path:out1:RootOutput(write)                0.010016      0.105189      0.509399      0.0880411     0.0814318       50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6137.74 MB
  Peak resident set size usage (VmHWM): 1028.15 MB
====================================================================================================
Art has completed and will exit with status 0.
'larcv.root' -> 'prod_monte-carlo-012686-000009_gen_g4_detsim_reco_larcv.root'
Sandbox folder contains:
total 277M
-rw-r--r-- 1 dune090 dune   60 Feb  3 17:37 all-input-dids.txt
-rw-r--r-- 1 dune090 dune    0 Feb  3 17:37 debugprod.log
drwxr-xr-x 2 dune090 dune 4.0K Jan 17 02:52 fhicl
-rw-r--r-- 1 dune090 dune  12K Feb  3 17:37 fhicl_bundle.tgz
-rw-r--r-- 1 dune090 dune  90K Feb  3 17:39 jobscript.log
-rw-r--r-- 1 dune090 dune 251M Feb  3 17:39 prod_monte-carlo-012686-000009_gen_g4_detsim_reco.root
-rw-r--r-- 1 dune090 dune  26M Feb  3 17:39 prod_monte-carlo-012686-000009_gen_g4_detsim_reco_larcv.root
-rw-r--r-- 1 dune090 dune  519 Feb  3 17:39 supera_hist.root
Wrote processed DID: usertests:prod_monte-carlo-012686-000009_gen_g4_detsim.root
=== RECOjob completed successfully ===