Jobsub ID 303562.0@dunegpschedd01.fnal.gov

Jobsub ID	303562.0@dunegpschedd01.fnal.gov
Workflow ID	12694
Stage ID	4
User name	usher@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2026-02-03 19:24:14
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2026-02-03 19:30:17
From worker node	Hostname	wn2208251.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7513 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-02-03 19:24:37
Input files	usertests:prod_monte-carlo-012694-000010_gen_g4_detsim.root
Jobscript	Exit code	0
	Real time	2m (179s)
	CPU time	2m (161s = 89%)
	Max RSS bytes	1312157696 (1251 MiB)
Outputting started	2026-02-03 19:27:37
Output files	usertests:prod_monte-carlo-012694-000010_gen_g4_detsim_reco.root usertests:prod_monte-carlo-012694-000010_gen_g4_detsim_reco_larcv.root
Finished	2026-02-03 19:30:17
Saved logs	justin-logs:303562.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

d 3d index 7980

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 48th record. run: 210010 subRun: 0 event: 48 at 03-Feb-2026 19:27:30 GMT
++++>>>> total num hits: 3059, num free: 3059

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 779!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 14465
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 3938

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 49th record. run: 210010 subRun: 0 event: 49 at 03-Feb-2026 19:27:31 GMT
++++>>>> total num hits: 7391, num free: 7391

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1995!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 75177
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 8962

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 50th record. run: 210010 subRun: 0 event: 50 at 03-Feb-2026 19:27:34 GMT
++++>>>> total num hits: 1512, num free: 1512

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1306!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 16189
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 16
 ... due to invalid 2d index 0
 ... due to invalid 3d index 1521

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
03-Feb-2026 19:27:36 GMT  Closed output file "prod_monte-carlo-012694-000010_gen_g4_detsim_reco.root"
03-Feb-2026 19:27:36 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/3d/4d/prod_monte-carlo-012694-000010_gen_g4_detsim.root"
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_mpv_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing neutrino_mpv_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_mcst_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_masked_true2reco_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_index_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_masked_true_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_highE_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_lowE_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_leftover_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_pcluster_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_at_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_chi2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_occupancy_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_nhits_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_cryo_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tpc_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge0_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge1_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time0_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time1_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key0_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key1_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_sed_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_ghost_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_corrected_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_rescaled_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Closing output file

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.250717       2.27025       15.0492       1.90283       2.22907        50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000357031   0.00174348     0.0407523    0.000632087   0.00565555       50     
reco:gaushit:GausHitFinder                     0.0184484     0.0901353     0.218823      0.073993      0.0518794       50     
reco:cluster3d:Cluster3D                      0.00853093     0.194754       5.17752      0.0549563     0.727783        50     
reco:opdec:Deconvolution                       0.0472935     0.178176      0.328312      0.179038      0.0573066       50     
reco:ophitspe:OpHitFinderDeco                 0.00328751     0.0156574     0.0515811     0.0132328    0.00813673       50     
reco:opflash:OpFlashFinder                    0.000357962   0.00165388    0.00354666     0.0016932    0.000543918      50     
reco:opslicer:OpSlicer                         0.0143222     0.343933      0.750918       0.33622      0.202146        50     
reco:mcreco:MCReco                            0.00705849     0.310828       1.74442      0.200521      0.337817        50     
[art]:TriggerResults:TriggerResultInserter     1.62e-05     3.07175e-05   9.3084e-05    2.3975e-05    1.56395e-05      50     
end_path:superaMC:LArSoftSuperaDriver          0.066993      0.989984       7.91702      0.776002       1.15259        50     
end_path:out1:RootOutput                       3.948e-06    1.03718e-05   3.5546e-05    9.5225e-06    6.15584e-06      50     
end_path:out1:RootOutput(write)                0.0282181     0.142475      0.925088      0.122293       0.12264        50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19403.3 MB
  Peak resident set size usage (VmHWM): 1333.5 MB
====================================================================================================
Art has completed and will exit with status 0.
'larcv.root' -> 'prod_monte-carlo-012694-000010_gen_g4_detsim_reco_larcv.root'
Sandbox folder contains:
total 359M
-rw-r--r-- 1 dune01 dune01   60 Feb  3 19:24 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01    0 Feb  3 19:25 debugprod.log
drwxr-xr-x 2 dune01 dune01 4.0K Jan 17 01:52 fhicl
-rw-r--r-- 1 dune01 dune01  12K Feb  3 19:24 fhicl_bundle.tgz
-rw-r--r-- 1 dune01 dune01  90K Feb  3 19:27 jobscript.log
-rw-r--r-- 1 dune01 dune01 326M Feb  3 19:27 prod_monte-carlo-012694-000010_gen_g4_detsim_reco.root
-rw-r--r-- 1 dune01 dune01  34M Feb  3 19:27 prod_monte-carlo-012694-000010_gen_g4_detsim_reco_larcv.root
-rw-r--r-- 1 dune01 dune01  519 Feb  3 19:27 supera_hist.root
Wrote processed DID: usertests:prod_monte-carlo-012694-000010_gen_g4_detsim.root
=== RECOjob completed successfully ===