justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 17996.14@dunegpschedd02.fnal.gov

Jobsub ID17996.14@dunegpschedd02.fnal.gov
Workflow ID1035
Stage ID1
User nameayankele@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-16 01:19:25
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-08-16 01:36:59
From worker nodeHostnamemilan22.farm.particle.cz
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-16 01:20:41
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_564_20231203T175940Z_gen_g4_detsim_hitreco__20240509T215922Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_942_20231203T235343Z_gen_g4_detsim_hitreco__20240509T211758Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_192_20231204T120351Z_gen_g4_detsim_hitreco__20240509T215630Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_616_20231204T052152Z_gen_g4_detsim_hitreco__20240509T213447Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_423_20231203T092851Z_gen_g4_detsim_hitreco__20240509T195214Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_500_20231208T093038Z_gen_g4_detsim_hitreco__20240510T063926Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_784_20231207T200705Z_gen_g4_detsim_hitreco__20240510T032408Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_186_20231201T075501Z_gen_g4_detsim_hitreco__20240508T070232Z_reco2.root
JobscriptExit code0
Real time15m (922s)
CPU time13m (814s = 88%)
Max RSS bytes2940010496 (2803 MiB)
Outputting started2025-08-16 01:36:04
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_564_20231203T175940Z_gen_g4_detsim_hitreco__20240509T215922Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_784_20231207T200705Z_gen_g4_detsim_hitreco__20240510T032408Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_500_20231208T093038Z_gen_g4_detsim_hitreco__20240510T063926Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_616_20231204T052152Z_gen_g4_detsim_hitreco__20240509T213447Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_423_20231203T092851Z_gen_g4_detsim_hitreco__20240509T195214Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_192_20231204T120351Z_gen_g4_detsim_hitreco__20240509T215630Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_942_20231203T235343Z_gen_g4_detsim_hitreco__20240509T211758Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_186_20231201T075501Z_gen_g4_detsim_hitreco__20240508T070232Z_reco2_cvnpreprocess.root
Finished2025-08-16 01:36:59
Saved logsjustin-logs:17996.14-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

99c94f/preprocess_atmnu/localProducts_larsoft_v10_03_01_e26_prof
----------- this block should be empty ------------------
WARNING: Unable to open file /cvmfs/fifeuser4.opensciencegrid.org/sw/dune/7610af3dffce908cd92377912a9d2dbbc399c94f/preprocess_atmnu/localProducts_larsoft_v10_03_01_e26_prof.
---------------------------------------------------------
ln: failed to create symbolic link './CCNuSelection.fcl': File exists
ln: failed to create symbolic link './CTPEvaluator.fcl': File exists
ln: failed to create symbolic link './CTPTrackDump.fcl': File exists
ln: failed to create symbolic link './CVNCreateLMDB.fcl': File exists
ln: failed to create symbolic link './CVNCreateLevelDB.fcl': File exists
ln: failed to create symbolic link './CVNCreateLevelDBTest.fcl': File exists
ln: failed to create symbolic link './CVNCreateZlibImages.fcl': File exists
ln: failed to create symbolic link './CVNEvaluator.fcl': File exists
ln: failed to create symbolic link './CVNEventDump.fcl': File exists
ln: failed to create symbolic link './CVNMapper.fcl': File exists
ln: failed to create symbolic link './CVNProngCreateLevelDB.fcl': File exists
ln: failed to create symbolic link './CVNSparseMapper.fcl': File exists
ln: failed to create symbolic link './CVNTrain.fcl': File exists
ln: failed to create symbolic link './CVNValidation.fcl': File exists
ln: failed to create symbolic link './CVNZlibMaker.fcl': File exists
ln: failed to create symbolic link './GCNGraphMaker.fcl': File exists
ln: failed to create symbolic link './GCNZlibMaker.fcl': File exists
ln: failed to create symbolic link './InfillChannelsConfig.fcl': File exists
ln: failed to create symbolic link './MakeInfillTrainingDataConfig.fcl': File exists
ln: failed to create symbolic link './RegCNNAna.fcl': File exists
ln: failed to create symbolic link './RegCNNEvaluator.fcl': File exists
ln: failed to create symbolic link './RegCNNEventDump.fcl': File exists
ln: failed to create symbolic link './RegCNNMapper.fcl': File exists
ln: failed to create symbolic link './RegCNNPyTorch.fcl': File exists
ln: failed to create symbolic link './RegCNNVertexAna.fcl': File exists
ln: failed to create symbolic link './VLNEnergyDataGenConfigs.fcl': File exists
ln: failed to create symbolic link './VLNEvaluatorConfigs.fcl': File exists
ln: failed to create symbolic link './VLNVarConfigs.fcl': File exists
ln: failed to create symbolic link './angularreco.fcl': File exists
ln: failed to create symbolic link './checkrecoenergy.fcl': File exists
ln: failed to create symbolic link './cluster3dalgorithms_dune35t.fcl': File exists
ln: failed to create symbolic link './clusteralgorithms_dune35t.fcl': File exists
ln: failed to create symbolic link './clusterfinder_dune.fcl': File exists
ln: failed to create symbolic link './ctphelper.fcl': File exists
ln: failed to create symbolic link './cvn_dune.fcl': File exists
ln: failed to create symbolic link './cvnmapevaluatejob.fcl': File exists
ln: failed to create symbolic link './cvnmapeventdumpjob.fcl': File exists
ln: failed to create symbolic link './disambigcheck.fcl': File exists
ln: failed to create symbolic link './dune_cvn_eval.fcl': File exists
ln: failed to create symbolic link './dune_cvn_event_dump_fhc.fcl': File exists
ln: failed to create symbolic link './dune_cvn_event_dump_rhc.fcl': File exists
ln: failed to create symbolic link './dune_cvn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './dunehd_1x2x6_cvn_eval_fhc.fcl': File exists
ln: failed to create symbolic link './dunehd_1x2x6_cvn_eval_rhc.fcl': File exists
ln: failed to create symbolic link './dunehd_1x2x6_cvn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './dunevd_3view_cvn_eval.fcl': File exists
ln: failed to create symbolic link './dunevd_3view_cvn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './dunevd_8x6_cvn_eval_fhc.fcl': File exists
ln: failed to create symbolic link './dunevd_8x6_cvn_eval_rhc.fcl': File exists
ln: failed to create symbolic link './dunevd_8x6_cvn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './energyreco.fcl': File exists
ln: failed to create symbolic link './hitfindermodules_dune.fcl': File exists
ln: failed to create symbolic link './multitpcdump.fcl': File exists
ln: failed to create symbolic link './mvamodule.fcl': File exists
ln: failed to create symbolic link './mvaselectnue.fcl': File exists
ln: failed to create symbolic link './mvaselectnumu.fcl': File exists
ln: failed to create symbolic link './neutrinoangularrecoalg_dune.fcl': File exists
ln: failed to create symbolic link './neutrinoenergyrecoalg_dune.fcl': File exists
ln: failed to create symbolic link './nueanamodule.fcl': File exists
ln: failed to create symbolic link './numberofhitsfilter.fcl': File exists
ln: failed to create symbolic link './pandoramodularshowertools_dune.fcl': File exists
ln: failed to create symbolic link './pandoramodules_dune.fcl': File exists
ln: failed to create symbolic link './pandoranuselection.fcl': File exists
ln: failed to create symbolic link './pdhd_wirecell_sim_deposplat.fcl': File exists
ln: failed to create symbolic link './pfpeff.fcl': File exists
ln: failed to create symbolic link './pointrestree.fcl': File exists
ln: failed to create symbolic link './pointrestree_1x2x2.fcl': File exists
ln: failed to create symbolic link './pointrestree_NuEScatter.fcl': File exists
ln: failed to create symbolic link './pointrestree_legacy.fcl': File exists
ln: failed to create symbolic link './pointrestree_marley.fcl': File exists
ln: failed to create symbolic link './protodune_cvn_eval.fcl': File exists
ln: failed to create symbolic link './protodune_cvn_event_dump.fcl': File exists
ln: failed to create symbolic link './protodune_gcn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './protodunevd_wirecell_img.fcl': File exists
ln: failed to create symbolic link './protodunevd_wirecell_sim_deposplat.fcl': File exists
ln: failed to create symbolic link './recoenergys.fcl': File exists
ln: failed to create symbolic link './recoenergys_v1.fcl': File exists
ln: failed to create symbolic link './recoenergys_vd.fcl': File exists
ln: failed to create symbolic link './recoenergystudies.fcl': File exists
ln: failed to create symbolic link './regcnn_numu_ana_job.fcl': File exists
ln: failed to create symbolic link './regcnn_numu_eval_job.fcl': File exists
ln: failed to create symbolic link './regcnnanajob.fcl': File exists
ln: failed to create symbolic link './regcnnevaluatejob.fcl': File exists
ln: failed to create symbolic link './regcnneventdumpjob.fcl': File exists
ln: failed to create symbolic link './runConvTrackDump.fcl': File exists
ln: failed to create symbolic link './runConvTrackPID.fcl': File exists
ln: failed to create symbolic link './runPandoraNuSelection_standard.fcl': File exists
ln: failed to create symbolic link './run_InfillChannelsFD.fcl': File exists
ln: failed to create symbolic link './run_InfillChannelsPD.fcl': File exists
ln: failed to create symbolic link './run_MakeInfillTrainingData.fcl': File exists
ln: failed to create symbolic link './runcheckereco.fcl': File exists
ln: failed to create symbolic link './sp_fd_adcdump_job_example.fcl': File exists
ln: failed to create symbolic link './trackfindermodules_dune.fcl': File exists
ln: failed to create symbolic link './vlnenergyanalyzejob.fcl': File exists
ln: failed to create symbolic link './vlnenergydatagenjob.fcl': File exists
ln: failed to create symbolic link './vlnenergyevalnumujob.fcl': File exists
ln: failed to create symbolic link './wirecell_dune.fcl': File exists
=== Start last 20 lines of lar log file ===
16-Aug-2025 03:35:58 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/42/c4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_186_20231201T075501Z_gen_g4_detsim_hitreco__20240508T070232Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.10344      0.705014       9.26852      0.322797       1.17604        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000846502    0.0105279     0.040142      0.0103697    0.00901967       100    
reco:rns:RandomNumberSaver                     4.182e-05    7.73208e-05   0.000733391   6.16605e-05   8.52829e-05      100    
[art]:TriggerResults:TriggerResultInserter     1.17e-05     1.9658e-05     6.155e-05    1.7515e-05    8.1816e-06       100    
end_path:recoEnergy:RecoEnergyS                0.100423      0.694168       9.26568      0.308525       1.1768         100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20800.8 MB
  Peak resident set size usage (VmHWM): 2375.84 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 20 lines of lar log file ===
lar exit code 0
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
justIN time: 2025-08-16 14:32:07 UTC       justIN version: 01.04.01