Jobsub ID 297627.13@dunegpschedd02.fnal.gov

Jobsub ID	297627.13@dunegpschedd02.fnal.gov
Workflow ID	12664
Stage ID	4
User name	nivedpm@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2026-02-03 10:26:43
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod02
Last heartbeat	2026-02-03 10:33:45
From worker node	Hostname	wm23.grid.hep.ph.ic.ac.uk
	cpuinfo	AMD EPYC 7742 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-02-03 10:27:42
Input files	usertests:prod_monte-carlo-012664-000004_gen_g4_detsim.root
Jobscript	Exit code	0
	Real time	3m (204s)
	CPU time	2m (169s = 82%)
	Max RSS bytes	1155702784 (1102 MiB)
Outputting started	2026-02-03 10:31:07
Output files	usertests:prod_monte-carlo-012664-000004_gen_g4_detsim_reco.root usertests:prod_monte-carlo-012664-000004_gen_g4_detsim_reco_larcv.root
Finished	2026-02-03 10:33:45
Saved logs	justin-logs:297627.13-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3d index 17827

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 48th record. run: 210004 subRun: 0 event: 48 at 03-Feb-2026 10:31:01 GMT
++++>>>> total num hits: 11270, num free: 11270

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1576!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 19070
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 334
 ... due to invalid 2d index 0
 ... due to invalid 3d index 3324

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 49th record. run: 210004 subRun: 0 event: 49 at 03-Feb-2026 10:31:02 GMT
++++>>>> total num hits: 74, num free: 74

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 175!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 2648
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 86

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 50th record. run: 210004 subRun: 0 event: 50 at 03-Feb-2026 10:31:03 GMT
++++>>>> total num hits: 1500, num free: 1500

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 977!
Filling voxel information...
   [93m[WARNING][00m [0m [94m<SuperaMCParticleCluster::AnalyzeSimChannel>[00m Total IDE ctr 14609
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 1063

    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m Saved interaction count: 1
    [95m[NORMAL][00m [0m [94m<SuperaMCParticleCluster::process>[00m 0 particles have no ancestor, summed deposit energy = 0
03-Feb-2026 10:31:06 GMT  Closed output file "prod_monte-carlo-012664-000004_gen_g4_detsim_reco.root"
03-Feb-2026 10:31:06 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/60/65/prod_monte-carlo-012664-000004_gen_g4_detsim.root"
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_mpv_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing neutrino_mpv_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_mcst_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_masked_true2reco_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_index_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_masked_true_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_highE_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_pcluster_lowE_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_leftover_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_pcluster_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing cluster3d_sed_at_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_chi2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_occupancy_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_nhits_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_cryo_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_tpc_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge0_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge1_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_charge2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time0_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time1_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_time2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key0_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key1_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_hit_key2_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_sed_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_pcluster_semantics_ghost_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing particle_corrected_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Writing sparse3d_reco_rescaled_tree with 50 entries
    [95m[NORMAL][00m [0m [94m<IOManager::finalize>[00m Closing output file

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.130627       2.40964       14.6698       1.70743       2.63283        50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000365579    0.0011943    0.00606856    0.000574739   0.00147336       50     
reco:gaushit:GausHitFinder                     0.028146      0.0913983     0.263504      0.0703902     0.0545204       50     
reco:cluster3d:Cluster3D                      0.00161205      0.2711        5.46748      0.0432579     0.975826        50     
reco:opdec:Deconvolution                       0.0157147      0.18668      0.333684      0.180993      0.0788925       50     
reco:ophitspe:OpHitFinderDeco                 0.00985113     0.0172584     0.0251637     0.0169653    0.00343583       50     
reco:opflash:OpFlashFinder                    0.000214925   0.00146038    0.00406945    0.00142049    0.000729337      50     
reco:opslicer:OpSlicer                        0.00018043     0.303247       0.99985      0.224907      0.264393        50     
reco:mcreco:MCReco                             0.0121019     0.284102       1.37145      0.145595      0.318368        50     
[art]:TriggerResults:TriggerResultInserter    1.1291e-05    2.59343e-05   0.000100991   2.26475e-05   1.55247e-05      50     
end_path:superaMC:LArSoftSuperaDriver          0.055165       1.09281       6.93621       0.76993       1.20166        50     
end_path:out1:RootOutput                       4.969e-06    6.32986e-06   1.4948e-05     5.911e-06    1.59296e-06      50     
end_path:out1:RootOutput(write)               0.00667551     0.159528       0.93264      0.124482      0.149428        50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 36343.4 MB
  Peak resident set size usage (VmHWM): 1155.7 MB
====================================================================================================
Art has completed and will exit with status 0.
'larcv.root' -> 'prod_monte-carlo-012664-000004_gen_g4_detsim_reco_larcv.root'
Sandbox folder contains:
total 294M
-rw-r--r--. 1 job0037 job0037   60 Feb  3 10:27 all-input-dids.txt
-rw-r--r--. 1 job0037 job0037    0 Feb  3 10:28 debugprod.log
drwxr-xr-x. 2 job0037 job0037 4.0K Jan 17 01:52 fhicl
-rw-r--r--. 1 job0037 job0037  12K Feb  3 10:27 fhicl_bundle.tgz
-rw-r--r--. 1 job0037 job0037  90K Feb  3 10:31 jobscript.log
-rw-r--r--. 1 job0037 job0037 265M Feb  3 10:31 prod_monte-carlo-012664-000004_gen_g4_detsim_reco.root
-rw-r--r--. 1 job0037 job0037  29M Feb  3 10:31 prod_monte-carlo-012664-000004_gen_g4_detsim_reco_larcv.root
-rw-r--r--. 1 job0037 job0037  519 Feb  3 10:31 supera_hist.root
Wrote processed DID: usertests:prod_monte-carlo-012664-000004_gen_g4_detsim.root
=== RECOjob completed successfully ===