justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 279313.57@dunegpschedd01.fnal.gov

Jobsub ID279313.57@dunegpschedd01.fnal.gov
Workflow ID11977
Stage ID1
User nameusher@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit14400 (4 hours)
Submitted time2026-01-15 21:20:19
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2026-01-15 22:21:00
From worker nodeHostnamewn-da-22.gina.surf.nl
cpuinfoAMD EPYC-Rome Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2026-01-15 21:21:16
Input filesmonte-carlo-011977-000019
JobscriptExit code0
Real time0m (20s)
CPU time0m (14s = 69%)
Max RSS bytes588800000 (561 MiB)
Outputting started2026-01-15 21:21:37
Output files
Finished2026-01-15 22:21:00
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eneration or simulation step.
%MSG
%MSG-w ParticleInventory:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 18
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w BackTrackerService:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 18
Rebuild failed to get the SimChannels. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 18
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  MultiPartVertex:generator@BeginModule  15-Jan-2026 22:21:34 CET run: 200019 subRun: 0 event: 18
Random seed for this event, engine 'generator.GenVertex': 691402667
%MSG
Processing a new event...
Generating a rain particle at (-168.52,131.94,1277.74)
Generated boost parameter gamma beta = 2.71069, cos(theta) = -0.642061, phi = 4.40891
Generating a boost (-0.214947,-0.686401,-0.602378)
Event Vertex @ (-168.52,131.94,1277.74) ... 10 particles...
  0th instance PDG 11
  1th instance PDG 111
  2th instance PDG 2212
  3th instance PDG 2212
  4th instance PDG 111
  5th instance PDG 2112
  6th instance PDG 2112
  7th instance PDG 2112
  8th instance PDG 2212
  9th instance PDG 2212
E: 0.457541 , m: 0.000510999 , KE: 0.45703
E: 0.48118 , m: 0.134977 , KE: 0.346203
E: 1.02767 , m: 0.938272 , KE: 0.0893938
E: 1.5677 , m: 0.938272 , KE: 0.629427
E: 0.245277 , m: 0.134977 , KE: 0.1103
E: 1.84811 , m: 0.939565 , KE: 0.908544
E: 1.17787 , m: 0.939565 , KE: 0.238308
E: 1.13232 , m: 0.939565 , KE: 0.192756
E: 1.468 , m: 0.938272 , KE: 0.529727
E: 1.48574 , m: 0.938272 , KE: 0.547466
total KE: 4.04916
KE range: 15 , pdg :  11
particle KE sf: 15, p : 0.457541 , mass :  0.000510999 , mom_sf: 14.9844
particle E : 0.457541 , scaled E : 6.85597
KE range: 2.5 , pdg :  111
particle KE sf: 2.5, p : 0.461861 , mass :  0.134977 , mom_sf: 2.1464
particle E : 0.48118 , scaled E : 1.00049
KE range: 2.5 , pdg :  111
particle KE sf: 2.5, p : 0.204798 , mass :  0.134977 , mom_sf: 1.89414
particle E : 0.245277 , scaled E : 0.410727
Total number particles: 0
Begin processing the 19th record. run: 200019 subRun: 0 event: 19 at 15-Jan-2026 22:21:34 CET
%MSG-w ParticleInventory:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 19
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 19
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 19
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w BackTrackerService:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 19
Rebuild failed to get the SimChannels. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 19
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  MultiPartVertex:generator@BeginModule  15-Jan-2026 22:21:34 CET run: 200019 subRun: 0 event: 19
Random seed for this event, engine 'generator.GenVertex': 207827100
%MSG
Processing a new event...
Generating a rain particle at (-29.4131,185.377,1311.33)
Generated boost parameter gamma beta = 0.766907, cos(theta) = -0.0777963, phi = 1.61955
Generating a boost (-0.0295684,0.605987,-0.0473431)
Event Vertex @ (-29.4131,185.377,1311.33) ... 1 particles...
  0th instance PDG 13
E: 0.864683 , m: 0.105658 , KE: 0.759025
total KE: 0.759025
gen_total_KE: 0.0288007 , KE scale factor :  0.0379443
KE range: 15 , pdg :  13
particle KE sf: 0.569165, p : 0.858203 , mass :  0.105658 , mom_sf: 0.614289
particle E : 0.864683 , scaled E : 0.537668
Total number particles: 0
Begin processing the 20th record. run: 200019 subRun: 0 event: 20 at 15-Jan-2026 22:21:34 CET
%MSG-w ParticleInventory:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 20
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 20
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 20
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w BackTrackerService:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 20
Rebuild failed to get the SimChannels. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 15-Jan-2026 22:21:34 CET  run: 200019 subRun: 0 event: 20
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  MultiPartVertex:generator@BeginModule  15-Jan-2026 22:21:34 CET run: 200019 subRun: 0 event: 20
Random seed for this event, engine 'generator.GenVertex': 854966702
%MSG
Processing a new event...
Generating a rain particle at (157.065,319.952,243.445)
Generated boost parameter gamma beta = 2.77944, cos(theta) = 0.839469, phi = 4.825
Generating a boost (0.0574565,-0.508082,0.7899)
Event Vertex @ (157.065,319.952,243.445) ... 7 particles...
  0th instance PDG 13
  1th instance PDG 2212
  2th instance PDG 3122
  3th instance PDG 22
  4th instance PDG -211
  5th instance PDG 111
  6th instance PDG 2112
E: 0.156298 , m: 0.105658 , KE: 0.0506401
E: 1.54575 , m: 0.938272 , KE: 0.607475
E: 1.85279 , m: 1.11568 , KE: 0.737108
E: 0.549812 , m: 0 , KE: 0.549812
E: 0.928428 , m: 0.13957 , KE: 0.788858
E: 0.257339 , m: 0.134977 , KE: 0.122362
E: 1.36694 , m: 0.939565 , KE: 0.427376
total KE: 3.28363
KE range: 15 , pdg :  13
particle KE sf: 15, p : 0.115176 , mass :  0.105658 , mom_sf: 7.45629
particle E : 0.156298 , scaled E : 0.86526
KE range: 3 , pdg :  3122
particle KE sf: 3, p : 1.47922 , mass :  1.11568 , mom_sf: 2.11893
particle E : 1.85279 , scaled E : 3.327
KE range: 2.5 , pdg :  22
particle KE sf: 2.5, p : 0.549812 , mass :  0 , mom_sf: 2.5
particle E : 0.549812 , scaled E : 1.37453
KE range: 2.5 , pdg :  -211
particle KE sf: 2.5, p : 0.917877 , mass :  0.13957 , mom_sf: 2.29562
particle E : 0.928428 , scaled E : 2.11171
KE range: 2.5 , pdg :  111
particle KE sf: 2.5, p : 0.2191 , mass :  0.134977 , mom_sf: 1.91562
particle E : 0.257339 , scaled E : 0.440883
Total number particles: 0
15-Jan-2026 22:21:36 CET  Closed output file "mpvmpr_monte-carlo-011977-000019_gen.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                    0.000295793   0.000680914    0.0050228    0.00042007    0.00100688       20     
--------------------------------------------------------------------------------------------------------------------------------
source:EmptyEvent(read)                        3.636e-05    5.87095e-05   0.00034629    3.88945e-05   6.71139e-05      20     
simulate:rns:RandomNumberSaver                1.0189e-05    2.43995e-05   0.000253063   1.12815e-05   5.25746e-05      20     
simulate:generator:MultiPartVertex            6.5615e-05    0.000171749   0.000984566   0.000132854    0.0001901       20     
[art]:TriggerResults:TriggerResultInserter     5.681e-06     7.619e-06    3.6369e-05     6.006e-06    6.60756e-06      20     
end_path:out1:RootOutput                       9.62e-07     1.73635e-06   1.2132e-05     1.143e-06    2.39603e-06      20     
end_path:out1:RootOutput(write)               4.0567e-05    0.000242619    0.0024873    5.5542e-05    0.000532547      20     
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 15-Jan-2026 22:21:36 CET  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
        (per event)           854966702   generator.GenVertex

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1336.57 MB
  Peak resident set size usage (VmHWM): 583.152 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 20 passed = 20 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         20         20          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 1.392234 Real = 2.463365

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1336.57 VmHWM = 583.152

Art has completed and will exit with status 0.
-rw-r--r--. 1 dune009 dune 107K Jan 15 22:21 mpvmpr_monte-carlo-011977-000019_gen.root
Input fcl file: mpvmpr_gen_1x2x6.fcl
genType: mpvmpr
Wrote processed DID to justin-processed-dids.txt: monte-carlo-011977-000019
=== GEN job completed successfully ===
justIN time: 2026-02-04 04:49:17 UTC       justIN version: 01.06.00