justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303339.1@dunegpschedd01.fnal.gov

Jobsub ID303339.1@dunegpschedd01.fnal.gov
Workflow ID12667
Stage ID4
User namescalvez@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-02-03 10:34:43
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel
Last heartbeat2026-02-03 10:39:43
From worker nodeHostnamewn-choc-005.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 10:36:09
Input filesusertests:prod_monte-carlo-012667-000004_gen_g4_detsim.root
JobscriptExit code0
Real time1m (81s)
CPU time1m (62s = 76%)
Max RSS bytes936267776 (892 MiB)
Outputting started2026-02-03 10:37:31
Output filesusertests:prod_monte-carlo-012667-000004_gen_g4_detsim_reco.root
usertests:prod_monte-carlo-012667-000004_gen_g4_detsim_reco_larcv.root
Finished2026-02-03 10:39:43
Saved logsjustin-logs:303339.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

to invalid 3d index 2836

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 8th record. run: 210004 subRun: 0 event: 8 at 03-Feb-2026 11:37:21 CET
++++>>>> total num hits: 140, num free: 140

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 570!
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 15504
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 253
 ... due to invalid 2d index 0
 ... due to invalid 3d index 6283

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 9th record. run: 210004 subRun: 0 event: 9 at 03-Feb-2026 11:37:22 CET
++++>>>> total num hits: 21602, num free: 21602

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1952!
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 43606
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 4511

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
Begin processing the 10th record. run: 210004 subRun: 0 event: 10 at 03-Feb-2026 11:37:26 CET
++++>>>> total num hits: 15581, num free: 15581

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2029!
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 43332
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 5694

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
03-Feb-2026 11:37:30 CET  Closed output file "prod_monte-carlo-012667-000004_gen_g4_detsim_reco.root"
03-Feb-2026 11:37:30 CET  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/e8/e5/prod_monte-carlo-012667-000004_gen_g4_detsim.root"
    [NORMAL]  <IOManager::finalize> Writing particle_mpv_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing neutrino_mpv_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_mcst_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_sed_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_masked_true2reco_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_index_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_masked_true_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_highE_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_lowE_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_leftover_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_semantics_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing particle_pcluster_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_sed_at_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_chi2_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_occupancy_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_nhits_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_cryo_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_tpc_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_charge0_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_charge1_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_charge2_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_time0_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_time1_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_time2_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_key0_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_key1_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_hit_key2_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_sed_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_semantics_ghost_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing particle_corrected_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_rescaled_tree with 10 entries
    [NORMAL]  <IOManager::finalize> Closing output file

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.67197       1.87385       3.69296       1.62973      0.955594        10     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000238304    0.0132163     0.113214     0.000331376    0.0336686       10     
reco:gaushit:GausHitFinder                     0.0620645     0.111015       0.19278      0.101124      0.038586        10     
reco:cluster3d:Cluster3D                       0.0035609     0.0797963     0.269485      0.0392576     0.0875371       10     
reco:opdec:Deconvolution                       0.116373      0.214032      0.283681      0.235098      0.0549375       10     
reco:ophitspe:OpHitFinderDeco                  0.0425558     0.0510163     0.0644515     0.0505379    0.00551225       10     
reco:opflash:OpFlashFinder                    0.000700414   0.00145485    0.00295626    0.00140726    0.000614725      10     
reco:opslicer:OpSlicer                         0.0292448     0.259063      0.609506      0.295803       0.16662        10     
reco:mcreco:MCReco                             0.0568326      0.26789      0.733375      0.207221      0.205295        10     
[art]:TriggerResults:TriggerResultInserter    1.6576e-05    2.36094e-05   6.4182e-05    1.96765e-05   1.36021e-05      10     
end_path:superaMC:LArSoftSuperaDriver          0.232056      0.768012       1.52952      0.679569      0.431959        10     
end_path:out1:RootOutput                       4.405e-06    5.5807e-06     1.16e-05     4.7015e-06    2.17502e-06      10     
end_path:out1:RootOutput(write)                0.0392634     0.107671      0.182067       0.10121      0.0469045       10     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4846.33 MB
  Peak resident set size usage (VmHWM): 936.268 MB
====================================================================================================
Art has completed and will exit with status 0.
'larcv.root' -> 'prod_monte-carlo-012667-000004_gen_g4_detsim_reco_larcv.root'
Sandbox folder contains:
total 66M
-rw-r--r-- 1 dune090 dune   60 Feb  3 11:36 all-input-dids.txt
-rw-r--r-- 1 dune090 dune    0 Feb  3 11:36 debugprod.log
drwxr-xr-x 2 dune090 dune 4.0K Jan 17 02:52 fhicl
-rw-r--r-- 1 dune090 dune  12K Feb  3 11:36 fhicl_bundle.tgz
-rw-r--r-- 1 dune090 dune  62K Feb  3 11:37 jobscript.log
-rw-r--r-- 1 dune090 dune  61M Feb  3 11:37 prod_monte-carlo-012667-000004_gen_g4_detsim_reco.root
-rw-r--r-- 1 dune090 dune 4.7M Feb  3 11:37 prod_monte-carlo-012667-000004_gen_g4_detsim_reco_larcv.root
-rw-r--r-- 1 dune090 dune  519 Feb  3 11:37 supera_hist.root
Wrote processed DID: usertests:prod_monte-carlo-012667-000004_gen_g4_detsim.root
=== RECOjob completed successfully ===
justIN time: 2026-02-04 07:19:39 UTC       justIN version: 01.06.00