justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 299900.0@dunegpschedd01.fnal.gov

Jobsub ID299900.0@dunegpschedd01.fnal.gov
Workflow ID12302
Stage ID1
User namemuditk@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4718592000 (4500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-23 11:36:15
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2026-01-23 12:03:14
From worker nodeHostnamedunegli-8405471-0-fnpc19116.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4718592000 (4500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2026-01-23 11:37:47
Input filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2.root
JobscriptExit code17
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2026-01-23 12:03:14
Saved logsjustin-logs:299900.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

saving raw::RawDigits tagged "daq"
Begin processing the 5th record. run: 5082 subRun: 0 event: 227905 at 23-Jan-2026 11:58:01 UTC
SimDepoSource got 141617 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 141617 depos spanning: [-2245.93, 8.61693e+22]us
Retagger: tagging trace set: daq with 30720 traces, 0 summary
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 95.509964 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 94.74996 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 93.84997 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 92.910034 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 90.46002 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 87.98995 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 87.829956 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 83.32999 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 80.07004 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 77.98974 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 76.51986 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 74.939766 sec
[12:02:40.814] I [ timer  ] Timer: wcls::SimChannelSink : 13.950224 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Drifter : 5.93004 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 5.150011 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 5.0700097 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 5.0700097 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 4.9900193 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 4.9600077 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 4.950009 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 4.940007 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 4.9200063 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 4.8300056 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 4.7700043 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 4.440003 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 4.2399983 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Digitizer : 4.149998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.949998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.879998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.769998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.6699982 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.669998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.669998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.6299982 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.619998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.619998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.599998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.559998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 3.499998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.9999979 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 2.9699976 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.9099977 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.8899977 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.8799977 sec
[12:02:40.814] I [ timer  ] Timer: wcls::FrameSaver : 2.8599977 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.849998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.8199978 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.809998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.7899978 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 2.7499979 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.7299979 sec
[12:02:40.814] I [ timer  ] Timer: wcls::SimDepoSource : 2.669998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.669998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Retagger : 2.629998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.599998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 2.499998 sec
[12:02:40.814] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.3899984 sec
[12:02:40.814] I [ timer  ] Timer: Total node execution : 1212.3695375919342 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
23-Jan-2026 12:02:53 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2_detsim_2026-01-23T_113806Z.root"
23-Jan-2026 12:02:53 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/8c/9e/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      277.019       285.589       298.844       283.83        8.15862         5     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00196219    0.00540965    0.00736452    0.00602857    0.00184714        5     
simulate:tpcrawdecoder:WireCellToolkit          267.55        275.873       288.781       274.489       8.0736          5     
simulate:rns:RandomNumberSaver                 6.345e-05    0.000196116    0.0004926     6.988e-05    0.000171764       5     
[art]:TriggerResults:TriggerResultInserter     2.654e-05    3.5564e-05     4.879e-05     3.366e-05    7.29426e-06       5     
end_path:out1:RootOutput                       3.93e-06      8.114e-06     2.047e-05     5.76e-06     6.22011e-06       5     
end_path:out1:RootOutput(write)                 9.33472       9.71032       10.1126       9.58312      0.317448         5     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5911.7 MB
  Peak resident set size usage (VmHWM): 4292.71 MB
====================================================================================================
ProvidedPedestalAdditionService::dtor: Deleting random engine with seed 263079513
ExponentialChannelNoiseService::dtor: Deleting random engine with seed 507255280
StuckBitAdcDistortionService:dtor: Deleting random engine with seed 464194836
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
htgettoken: Failure getting token from https://htvaultprod.fnal.gov:8200

Fri Jan 23 12:02:54 2026
 Notice: Unable to find valid grid or kerberos credentials. Later actions will likely fail.
Copying 964073572 bytes file:///home/workspace/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2_detsim_2026-01-23T_113806Z.root => https://fndcadoor.fnal.gov:2880/dune/scratch/users/muditk/SN_test//prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2_detsim_2026-01-23T_113806Z.root
gfal-copy error: 17 (File exists) - TRANSFER ERROR: Copy failed (streamed). Last attempt: HTTP 403 : Permission refused  (destination)
Perhaps you forgot a -D to indicate destination is a directory? 
Fri Jan 23 12:02:55 2026
 program: www_cp.sh  /home/workspace/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2_detsim_2026-01-23T_113806Z.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/muditk/SN_test//prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2_detsim_2026-01-23T_113806Z.rootexited status 17
delaying 1 ...
htgettoken: Failure getting token from https://htvaultprod.fnal.gov:8200

Fri Jan 23 12:02:56 2026
 Notice: Unable to find valid grid or kerberos credentials. Later actions will likely fail.
Fri Jan 23 12:02:56 2026
 retrying...
Copying 964073572 bytes file:///home/workspace/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2_detsim_2026-01-23T_113806Z.root => https://fndcadoor.fnal.gov:2880/dune/scratch/users/muditk/SN_test//prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2_detsim_2026-01-23T_113806Z.root
gfal-copy error: 17 (File exists) - TRANSFER ERROR: Copy failed (streamed). Last attempt: HTTP 403 : Permission refused  (destination)
Perhaps you forgot a -D to indicate destination is a directory? 
Error during output copyback. See output logs.
justIN time: 2026-02-05 02:56:58 UTC       justIN version: 01.06.00