justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303340.0@dunegpschedd01.fnal.gov

Jobsub ID303340.0@dunegpschedd01.fnal.gov
Workflow ID12664
Stage ID4
User namenivedpm@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-02-03 10:38:44
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-02-03 10:57:48
From worker nodeHostnamecn053.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 10:40:03
Input filesusertests:prod_monte-carlo-012664-000005_gen_g4_detsim.root
JobscriptExit code0
Real time9m (558s)
CPU time2m (179s = 32%)
Max RSS bytes1166946304 (1112 MiB)
Outputting started2026-02-03 10:49:22
Output filesusertests:prod_monte-carlo-012664-000005_gen_g4_detsim_reco.root
usertests:prod_monte-carlo-012664-000005_gen_g4_detsim_reco_larcv.root
Finished2026-02-03 10:57:48
Saved logsjustin-logs:303340.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

d index 1709

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 48th record. run: 210005 subRun: 0 event: 48 at 03-Feb-2026 10:48:41 GMT
++++>>>> total num hits: 13509, num free: 13509

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2145!
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 64374
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 14452

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 49th record. run: 210005 subRun: 0 event: 49 at 03-Feb-2026 10:48:53 GMT
++++>>>> total num hits: 4395, num free: 4395

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1478!
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 71050
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 3709

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 50th record. run: 210005 subRun: 0 event: 50 at 03-Feb-2026 10:49:08 GMT
++++>>>> total num hits: 2678, num free: 2678

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1988!
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 71982
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 7955

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
03-Feb-2026 10:49:21 GMT  Closed output file "prod_monte-carlo-012664-000005_gen_g4_detsim_reco.root"
03-Feb-2026 10:49:21 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/c6/dd/prod_monte-carlo-012664-000005_gen_g4_detsim.root"
    [NORMAL]  <IOManager::finalize> Writing particle_mpv_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing neutrino_mpv_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_mcst_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_sed_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_masked_true2reco_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_index_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_masked_true_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_highE_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_lowE_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_leftover_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_semantics_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing particle_pcluster_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_sed_at_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_chi2_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_occupancy_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_nhits_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_cryo_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_tpc_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_charge0_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_charge1_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_charge2_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_time0_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_time1_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_time2_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_key0_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_key1_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_key2_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_sed_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_semantics_ghost_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing particle_corrected_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_rescaled_tree with 50 entries
    [NORMAL]  <IOManager::finalize> Closing output file

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.313296       6.37542       20.0674       5.78422       3.61524        50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000294606    0.0421427      0.59842     0.000710736    0.125186        50     
reco:gaushit:GausHitFinder                     0.035957      0.712162       1.68179      0.744467      0.453686        50     
reco:cluster3d:Cluster3D                      0.00262409     0.322172       4.79917      0.0883726     0.814667        50     
reco:opdec:Deconvolution                       0.0577542     0.825431       2.20646      0.800821      0.507177        50     
reco:ophitspe:OpHitFinderDeco                 0.00636054     0.180683       1.63478      0.0138309     0.415675        50     
reco:opflash:OpFlashFinder                    0.000337842   0.00168969    0.00321522    0.00174056    0.000622421      50     
reco:opslicer:OpSlicer                        0.00686213     0.353705      0.979177       0.32135      0.247496        50     
reco:mcreco:MCReco                             0.0152573     0.939564       2.1792        1.00461      0.578381        50     
[art]:TriggerResults:TriggerResultInserter    1.5088e-05    2.74448e-05   0.000157139   2.3914e-05    2.07967e-05      50     
end_path:superaMC:LArSoftSuperaDriver          0.122509       2.84543       7.93015       2.46987       1.81768        50     
end_path:out1:RootOutput                       3.754e-06    5.71474e-06   1.5757e-05     5.459e-06    1.71654e-06      50     
end_path:out1:RootOutput(write)                0.0234615     0.151726      0.769449      0.126843      0.127728        50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14865 MB
  Peak resident set size usage (VmHWM): 1166.19 MB
====================================================================================================
Art has completed and will exit with status 0.
'larcv.root' -> 'prod_monte-carlo-012664-000005_gen_g4_detsim_reco_larcv.root'
Sandbox folder contains:
total 396M
-rw-r--r-- 1 pildune32 pildune   60 Feb  3 10:40 all-input-dids.txt
-rw-r--r-- 1 pildune32 pildune    0 Feb  3 10:40 debugprod.log
drwxr-xr-x 2 pildune32 pildune 4.0K Jan 17 01:52 fhicl
-rw-r--r-- 1 pildune32 pildune  12K Feb  3 10:40 fhicl_bundle.tgz
-rw-r--r-- 1 pildune32 pildune  90K Feb  3 10:49 jobscript.log
-rw-r--r-- 1 pildune32 pildune 351M Feb  3 10:49 prod_monte-carlo-012664-000005_gen_g4_detsim_reco.root
-rw-r--r-- 1 pildune32 pildune  45M Feb  3 10:49 prod_monte-carlo-012664-000005_gen_g4_detsim_reco_larcv.root
-rw-r--r-- 1 pildune32 pildune  519 Feb  3 10:49 supera_hist.root
Wrote processed DID: usertests:prod_monte-carlo-012664-000005_gen_g4_detsim.root
=== RECOjob completed successfully ===
justIN time: 2026-02-04 07:19:05 UTC       justIN version: 01.06.00