Jobsub ID 278641.4@dunegpschedd01.fnal.gov

Jobsub ID	278641.4@dunegpschedd01.fnal.gov
Workflow ID	11941
Stage ID	4
User name	usher@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	8388608000 (8000 MiB)
	Wall seconds limit	28800 (8 hours)
Submitted time	2026-01-15 02:53:21
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc02
Last heartbeat	2026-01-15 03:15:35
From worker node	Hostname	wn-lb-13.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	10485760000 (10000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-15 02:54:21
Input files	usertests:mpvmpr_monte-carlo-011941-000028_gen_g4_detsim.root
Jobscript	Exit code	0
	Real time	16m (969s)
	CPU time	15m (951s = 98%)
	Max RSS bytes	1840664576 (1755 MiB)
Outputting started	2026-01-15 03:10:30
Output files	usertests:mpvmpr_monte-carlo-011941-000028_gen_g4_detsim_reco.root usertests:mpvmpr_monte-carlo-011941-000028_gen_g4_detsim_reco_larcv.root
Finished	2026-01-15 03:15:35
Saved logs	justin-logs:278641.4-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0 particles have no ancestor, summed deposit energy = 0
Begin processing the 18th record. run: 200000 subRun: 0 event: 18 at 15-Jan-2026 04:07:14 CET
++++>>>> total num hits: 40986, num free: 40986

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3063!
bad particle: track id = 0 PDG = 16 StatusCode = 0
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 1534525
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 3736
 ... due to invalid 2d index 0
 ... due to invalid 3d index 799955

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 19th record. run: 200000 subRun: 0 event: 19 at 15-Jan-2026 04:09:58 CET
++++>>>> total num hits: 6372, num free: 6372

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3295!
bad particle: track id = 0 PDG = 16 StatusCode = 0
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 275561
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 10839
 ... due to invalid 2d index 0
 ... due to invalid 3d index 122516

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 20th record. run: 200000 subRun: 0 event: 20 at 15-Jan-2026 04:10:06 CET
++++>>>> total num hits: 12488, num free: 12488

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2969!
bad particle: track id = 0 PDG = 16 StatusCode = 0
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 628793
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 4029
 ... due to invalid 2d index 0
 ... due to invalid 3d index 315529

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
15-Jan-2026 04:10:29 CET  Closed output file "mpvmpr_monte-carlo-011941-000028_gen_g4_detsim_reco.root"
15-Jan-2026 04:10:29 CET  Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/58/33/mpvmpr_monte-carlo-011941-000028_gen_g4_detsim.root"
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_mpv_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing neutrino_mpv_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_mcst_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_masked_true2reco_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_index_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_masked_true_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_highE_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_lowE_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_leftover_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_pcluster_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_at_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_chi2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_occupancy_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_nhits_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_cryo_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tpc_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge0_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge1_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time0_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time1_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key0_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key1_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_sed_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_ghost_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_corrected_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_rescaled_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Closing output file

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      2.0549        44.9657       163.504       29.0471       48.3284        20     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000864623   0.00206586    0.00617983    0.00173125    0.00129767       20     
reco:gaushit:GausHitFinder                     0.0690079     0.221093      0.432304      0.211109      0.106265        20     
reco:cluster3d:Cluster3D                       0.0499425      4.45234       25.2671       1.74769       6.60885        20     
reco:opdec:Deconvolution                       0.158856      0.250179      0.316822      0.262049      0.0508314       20     
reco:ophitspe:OpHitFinderDeco                  0.011114      0.0142452     0.0166492     0.014381     0.00183185       20     
reco:opflash:OpFlashFinder                    0.00139358    0.00220015     0.0039329    0.00221348    0.00057362       20     
reco:opslicer:OpSlicer                         0.209617      0.504246       0.8056       0.500308       0.18942        20     
reco:mcreco:MCReco                             0.101396      0.930373       2.55809      0.996715      0.645732        20     
[art]:TriggerResults:TriggerResultInserter    2.2652e-05    3.64476e-05   0.000111927   3.00345e-05   2.11628e-05      20     
end_path:superaMC:LArSoftSuperaDriver           1.08454       38.3097       153.889       20.9722       45.5906        20     
end_path:out1:RootOutput                       7.171e-06    8.20375e-06   1.6044e-05     7.771e-06    1.82258e-06      20     
end_path:out1:RootOutput(write)                 0.10467      0.278207      0.608539       0.25451      0.124142        20     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18897.5 MB
  Peak resident set size usage (VmHWM): 1840.66 MB
====================================================================================================
Art has completed and will exit with status 0.
'larcv.root' -> 'mpvmpr_monte-carlo-011941-000028_gen_g4_detsim_reco_larcv.root'
Sandbox folder contains:
total 800M
-rw-r--r--. 1 dune009 dune   62 Jan 15 03:54 all-input-dids.txt
-rw-r--r--. 1 dune009 dune    0 Jan 15 03:54 debugprod.log
drwxr-xr-x. 2 dune009 dune 4.0K Jan 14 22:30 fhicl
-rw-r--r--. 1 dune009 dune  12K Jan 15 03:54 fhicl_bundle.tgz
-rw-r--r--. 1 dune009 dune  70K Jan 15 04:10 jobscript.log
-rw-r--r--. 1 dune009 dune 752M Jan 15 04:10 mpvmpr_monte-carlo-011941-000028_gen_g4_detsim_reco.root
-rw-r--r--. 1 dune009 dune  49M Jan 15 04:10 mpvmpr_monte-carlo-011941-000028_gen_g4_detsim_reco_larcv.root
-rw-r--r--. 1 dune009 dune  519 Jan 15 04:10 supera_hist.root
Wrote processed DID: usertests:mpvmpr_monte-carlo-011941-000028_gen_g4_detsim.root
=== RECOjob completed successfully ===