Jobsub ID 271826.0@dunegpschedd02.fnal.gov

Jobsub ID	271826.0@dunegpschedd02.fnal.gov
Workflow ID	11877
Stage ID	4
User name	usher@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	8388608000 (8000 MiB)
	Wall seconds limit	28800 (8 hours)
Submitted time	2026-01-14 05:36:12
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2026-01-14 05:39:01
From worker node	Hostname	hnode52.pic.es
	cpuinfo	AMD EPYC 7402P 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	10485760000 (10000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-14 05:37:01
Input files	usertests:prod_monte-carlo-011877-000003_gen_g4_detsim.root
Jobscript	Exit code	0
	Real time	1m (67s)
	CPU time	0m (50s = 74%)
	Max RSS bytes	936996864 (893 MiB)
Outputting started	2026-01-14 05:38:09
Output files	usertests:prod_monte-carlo-011877-000003_gen_g4_detsim_reco.root
Finished	2026-01-14 05:39:01
Saved logs	justin-logs:271826.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

d particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 6961

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 18th record. run: 20000014 subRun: 0 event: 18 at 14-Jan-2026 06:38:06 CET
++++>>>> total num hits: 311, num free: 311

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 622!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 5238
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 117

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 19th record. run: 20000014 subRun: 0 event: 19 at 14-Jan-2026 06:38:06 CET
++++>>>> total num hits: 189, num free: 189

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 658!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 19490
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 1301

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 20th record. run: 20000014 subRun: 0 event: 20 at 14-Jan-2026 06:38:06 CET
++++>>>> total num hits: 1274, num free: 1274

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2218!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 36091
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 10
 ... due to invalid 2d index 0
 ... due to invalid 3d index 2192

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
14-Jan-2026 06:38:09 CET  Closed output file "prod_monte-carlo-011877-000003_gen_g4_detsim_reco.root"
14-Jan-2026 06:38:09 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/usertests/0f/93/prod_monte-carlo-011877-000003_gen_g4_detsim.root"
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_mpv_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing neutrino_mpv_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_mcst_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_masked_true2reco_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_index_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_masked_true_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_highE_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_lowE_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_leftover_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_pcluster_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_at_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_chi2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_occupancy_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_nhits_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_cryo_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tpc_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge0_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge1_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time0_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time1_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key0_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key1_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key2_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_sed_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_ghost_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_corrected_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_rescaled_tree with 20 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Closing output file

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0855535     0.922855       2.44998       0.81367      0.641139        20     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000198807   0.000425801   0.00147967    0.000304472   0.000326794      20     
reco:gaushit:GausHitFinder                     0.0113327     0.0422857     0.0871184     0.0421616     0.0220241       20     
reco:cluster3d:Cluster3D                      0.00153755     0.0377566     0.246707      0.0145707     0.0548187       20     
reco:opdec:Deconvolution                       0.0202635     0.110228      0.188904      0.114215      0.0487164       20     
reco:ophitspe:OpHitFinderDeco                 0.00170159    0.00449384    0.00689836    0.00444405    0.00155823       20     
reco:opflash:OpFlashFinder                    0.000177257   0.000810419   0.00185048    0.000766514   0.000393776      20     
reco:opslicer:OpSlicer                        0.00110731     0.147405      0.393886      0.119722      0.123072        20     
reco:mcreco:MCReco                            0.00393841     0.149071      0.562982      0.0796555     0.162535        20     
[art]:TriggerResults:TriggerResultInserter       9e-06      1.75201e-05   6.0502e-05    1.53805e-05   1.03462e-05      20     
end_path:superaMC:LArSoftSuperaDriver          0.0241089     0.368541       1.00763       0.36781      0.261644        20     
end_path:out1:RootOutput                       2.901e-06    5.2594e-06     1.218e-05     4.986e-06    1.76402e-06      20     
end_path:out1:RootOutput(write)                0.0105535     0.0613019     0.118953      0.0581184     0.0301469       20     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8088.21 MB
  Peak resident set size usage (VmHWM): 936.997 MB
====================================================================================================
Art has completed and will exit with status 0.
-rw-r--r-- 1 dune001 dune 103M Jan 14 06:38 prod_monte-carlo-011877-000003_gen_g4_detsim_reco.root
Sandbox folder contains:
total 107M
-rw-r--r-- 1 dune001 dune   60 Jan 14 06:37 all-input-dids.txt
-rw-r--r-- 1 dune001 dune    0 Jan 14 06:37 debugprod.log
drwxr-xr-x 2 dune001 dune 4.0K Jan 13 21:41 fhicl
-rw-r--r-- 1 dune001 dune  12K Jan 14 06:37 fhicl_bundle.tgz
-rw-r--r-- 1 dune001 dune  69K Jan 14 06:38 jobscript.log
-rw-r--r-- 1 dune001 dune 4.0M Jan 14 06:38 larcv.root
-rw-r--r-- 1 dune001 dune 103M Jan 14 06:38 prod_monte-carlo-011877-000003_gen_g4_detsim_reco.root
-rw-r--r-- 1 dune001 dune  519 Jan 14 06:38 supera_hist.root
Wrote processed DID: usertests:prod_monte-carlo-011877-000003_gen_g4_detsim.root
=== RECOjob completed successfully ===