justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 17996.26@dunegpschedd02.fnal.gov

Jobsub ID17996.26@dunegpschedd02.fnal.gov
Workflow ID1035
Stage ID1
User nameayankele@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-16 01:19:25
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-08-16 01:37:47
From worker nodeHostnamelupin17.farm.particle.cz
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-16 01:21:42
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_44_20231120T132541Z_gen_g4_detsim_hitreco__20240503T062204Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_118_20231201T120817Z_gen_g4_detsim_hitreco__20240507T185815Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_154_20231201T220551Z_gen_g4_detsim_hitreco__20240507T203943Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1132_20231204T120526Z_gen_g4_detsim_hitreco__20240509T215256Z_reco2.root
JobscriptExit code0
Real time15m (909s)
CPU time12m (772s = 84%)
Max RSS bytes3224563712 (3075 MiB)
Outputting started2025-08-16 01:36:52
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_154_20231201T220551Z_gen_g4_detsim_hitreco__20240507T203943Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1132_20231204T120526Z_gen_g4_detsim_hitreco__20240509T215256Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_44_20231120T132541Z_gen_g4_detsim_hitreco__20240503T062204Z_reco2_cvnpreprocess.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/ayankele/fnal/01035/1/001/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_118_20231201T120817Z_gen_g4_detsim_hitreco__20240507T185815Z_reco2_cvnpreprocess.root
Finished2025-08-16 01:37:47
Saved logsjustin-logs:17996.26-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

77912a9d2dbbc399c94f/preprocess_atmnu/localProducts_larsoft_v10_03_01_e26_prof.
---------------------------------------------------------
ln: failed to create symbolic link './CCNuSelection.fcl': File exists
ln: failed to create symbolic link './CTPEvaluator.fcl': File exists
ln: failed to create symbolic link './CTPTrackDump.fcl': File exists
ln: failed to create symbolic link './CVNCreateLMDB.fcl': File exists
ln: failed to create symbolic link './CVNCreateLevelDB.fcl': File exists
ln: failed to create symbolic link './CVNCreateLevelDBTest.fcl': File exists
ln: failed to create symbolic link './CVNCreateZlibImages.fcl': File exists
ln: failed to create symbolic link './CVNEvaluator.fcl': File exists
ln: failed to create symbolic link './CVNEventDump.fcl': File exists
ln: failed to create symbolic link './CVNMapper.fcl': File exists
ln: failed to create symbolic link './CVNProngCreateLevelDB.fcl': File exists
ln: failed to create symbolic link './CVNSparseMapper.fcl': File exists
ln: failed to create symbolic link './CVNTrain.fcl': File exists
ln: failed to create symbolic link './CVNValidation.fcl': File exists
ln: failed to create symbolic link './CVNZlibMaker.fcl': File exists
ln: failed to create symbolic link './GCNGraphMaker.fcl': File exists
ln: failed to create symbolic link './GCNZlibMaker.fcl': File exists
ln: failed to create symbolic link './InfillChannelsConfig.fcl': File exists
ln: failed to create symbolic link './MakeInfillTrainingDataConfig.fcl': File exists
ln: failed to create symbolic link './RegCNNAna.fcl': File exists
ln: failed to create symbolic link './RegCNNEvaluator.fcl': File exists
ln: failed to create symbolic link './RegCNNEventDump.fcl': File exists
ln: failed to create symbolic link './RegCNNMapper.fcl': File exists
ln: failed to create symbolic link './RegCNNPyTorch.fcl': File exists
ln: failed to create symbolic link './RegCNNVertexAna.fcl': File exists
ln: failed to create symbolic link './VLNEnergyDataGenConfigs.fcl': File exists
ln: failed to create symbolic link './VLNEvaluatorConfigs.fcl': File exists
ln: failed to create symbolic link './VLNVarConfigs.fcl': File exists
ln: failed to create symbolic link './angularreco.fcl': File exists
ln: failed to create symbolic link './checkrecoenergy.fcl': File exists
ln: failed to create symbolic link './cluster3dalgorithms_dune35t.fcl': File exists
ln: failed to create symbolic link './clusteralgorithms_dune35t.fcl': File exists
ln: failed to create symbolic link './clusterfinder_dune.fcl': File exists
ln: failed to create symbolic link './ctphelper.fcl': File exists
ln: failed to create symbolic link './cvn_dune.fcl': File exists
ln: failed to create symbolic link './cvnmapevaluatejob.fcl': File exists
ln: failed to create symbolic link './cvnmapeventdumpjob.fcl': File exists
ln: failed to create symbolic link './disambigcheck.fcl': File exists
ln: failed to create symbolic link './dune_cvn_eval.fcl': File exists
ln: failed to create symbolic link './dune_cvn_event_dump_fhc.fcl': File exists
ln: failed to create symbolic link './dune_cvn_event_dump_rhc.fcl': File exists
ln: failed to create symbolic link './dune_cvn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './dunehd_1x2x6_cvn_eval_fhc.fcl': File exists
ln: failed to create symbolic link './dunehd_1x2x6_cvn_eval_rhc.fcl': File exists
ln: failed to create symbolic link './dunehd_1x2x6_cvn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './dunevd_3view_cvn_eval.fcl': File exists
ln: failed to create symbolic link './dunevd_3view_cvn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './dunevd_8x6_cvn_eval_fhc.fcl': File exists
ln: failed to create symbolic link './dunevd_8x6_cvn_eval_rhc.fcl': File exists
ln: failed to create symbolic link './dunevd_8x6_cvn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './energyreco.fcl': File exists
ln: failed to create symbolic link './hitfindermodules_dune.fcl': File exists
ln: failed to create symbolic link './multitpcdump.fcl': File exists
ln: failed to create symbolic link './mvamodule.fcl': File exists
ln: failed to create symbolic link './mvaselectnue.fcl': File exists
ln: failed to create symbolic link './mvaselectnumu.fcl': File exists
ln: failed to create symbolic link './neutrinoangularrecoalg_dune.fcl': File exists
ln: failed to create symbolic link './neutrinoenergyrecoalg_dune.fcl': File exists
ln: failed to create symbolic link './nueanamodule.fcl': File exists
ln: failed to create symbolic link './numberofhitsfilter.fcl': File exists
ln: failed to create symbolic link './pandoramodularshowertools_dune.fcl': File exists
ln: failed to create symbolic link './pandoramodules_dune.fcl': File exists
ln: failed to create symbolic link './pandoranuselection.fcl': File exists
ln: failed to create symbolic link './pdhd_wirecell_sim_deposplat.fcl': File exists
ln: failed to create symbolic link './pfpeff.fcl': File exists
ln: failed to create symbolic link './pointrestree.fcl': File exists
ln: failed to create symbolic link './pointrestree_1x2x2.fcl': File exists
ln: failed to create symbolic link './pointrestree_NuEScatter.fcl': File exists
ln: failed to create symbolic link './pointrestree_legacy.fcl': File exists
ln: failed to create symbolic link './pointrestree_marley.fcl': File exists
ln: failed to create symbolic link './protodune_cvn_eval.fcl': File exists
ln: failed to create symbolic link './protodune_cvn_event_dump.fcl': File exists
ln: failed to create symbolic link './protodune_gcn_zlib_maker.fcl': File exists
ln: failed to create symbolic link './protodunevd_wirecell_img.fcl': File exists
ln: failed to create symbolic link './protodunevd_wirecell_sim_deposplat.fcl': File exists
ln: failed to create symbolic link './recoenergys.fcl': File exists
ln: failed to create symbolic link './recoenergys_v1.fcl': File exists
ln: failed to create symbolic link './recoenergys_vd.fcl': File exists
ln: failed to create symbolic link './recoenergystudies.fcl': File exists
ln: failed to create symbolic link './regcnn_numu_ana_job.fcl': File exists
ln: failed to create symbolic link './regcnn_numu_eval_job.fcl': File exists
ln: failed to create symbolic link './regcnnanajob.fcl': File exists
ln: failed to create symbolic link './regcnnevaluatejob.fcl': File exists
ln: failed to create symbolic link './regcnneventdumpjob.fcl': File exists
ln: failed to create symbolic link './runConvTrackDump.fcl': File exists
ln: failed to create symbolic link './runConvTrackPID.fcl': File exists
ln: failed to create symbolic link './runPandoraNuSelection_standard.fcl': File exists
ln: failed to create symbolic link './run_InfillChannelsFD.fcl': File exists
ln: failed to create symbolic link './run_InfillChannelsPD.fcl': File exists
ln: failed to create symbolic link './run_MakeInfillTrainingData.fcl': File exists
ln: failed to create symbolic link './runcheckereco.fcl': File exists
ln: failed to create symbolic link './sp_fd_adcdump_job_example.fcl': File exists
ln: failed to create symbolic link './trackfindermodules_dune.fcl': File exists
ln: failed to create symbolic link './vlnenergyanalyzejob.fcl': File exists
ln: failed to create symbolic link './vlnenergydatagenjob.fcl': File exists
ln: failed to create symbolic link './vlnenergyevalnumujob.fcl': File exists
ln: failed to create symbolic link './wirecell_dune.fcl': File exists
=== Start last 20 lines of lar log file ===
16-Aug-2025 03:36:42 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7d/a3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1132_20231204T120526Z_gen_g4_detsim_hitreco__20240509T215256Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.174562      0.978069       8.33126      0.508856       1.38104        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00193557     0.0172515     0.064354      0.0149561     0.0124525       100    
reco:rns:RandomNumberSaver                     5.333e-05    8.1578e-05    0.00076537    6.8645e-05    7.18628e-05      100    
[art]:TriggerResults:TriggerResultInserter     1.439e-05    2.29603e-05   0.00011399    1.9765e-05    1.29087e-05      100    
end_path:recoEnergy:RecoEnergyS                0.170322      0.960424       8.32869      0.475359       1.3809         100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20413.3 MB
  Peak resident set size usage (VmHWM): 2000.58 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 20 lines of lar log file ===
lar exit code 0
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
justIN time: 2025-08-16 14:31:47 UTC       justIN version: 01.04.01