Jobsub ID 40628.0@dunegpschedd02.fnal.gov
| Jobsub ID | 40628.0@dunegpschedd02.fnal.gov |
| Workflow ID | 2573 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 8387559424 (7999 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-09-16 21:50:24 |
| Site | CZ_FZU |
| Entry | Nova_CZ_FZU_osgce2 |
| Last heartbeat | 2025-09-17 00:50:25 |
| From worker node | Hostname | mikan44.farm.particle.cz |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 10485760000 (10000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-16 21:51:21 |
| Input files | usertests:prod_beam_p5GeV_protodunehd_1_39241_12_1757750222_pdint_TPC1_pdint_TPC2_pdint_TPC5_pdint_TPC6.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (10433s) |
| CPU time | 2h (10071s = 96%) |
| Max RSS bytes | 7549120512 (7199 MiB) |
| Outputting started | 2025-09-17 00:45:15 |
| Output files | |
| Finished | 2025-09-17 00:50:25 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 90605 traces tagged "gauss"
FrameSaver: q=2.48644e+06 n=1349673 tag=gauss
wclsFrameSaver: saving 107417 traces tagged "wiener"
FrameSaver: q=2.65407e+06 n=1312547 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 85523 traces, 85523 summary
Retagger: tagging trace set: gauss with 72509 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72509 traces tagged "gauss"
FrameSaver: q=1.917e+06 n=985408 tag=gauss
wclsFrameSaver: saving 85523 traces tagged "wiener"
FrameSaver: q=2.03083e+06 n=977343 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 89284 traces, 89284 summary
Retagger: tagging trace set: gauss with 78188 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 78188 traces tagged "gauss"
FrameSaver: q=2.00981e+06 n=1099942 tag=gauss
wclsFrameSaver: saving 89284 traces tagged "wiener"
FrameSaver: q=2.16197e+06 n=1103267 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 113503 traces, 113503 summary
Retagger: tagging trace set: gauss with 96988 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 96988 traces tagged "gauss"
FrameSaver: q=2.85077e+06 n=1524443 tag=gauss
wclsFrameSaver: saving 113503 traces tagged "wiener"
FrameSaver: q=3.00914e+06 n=1477433 tag=wiener
17-Sep-2025 02:45:11 CEST Closed output file "prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1_reco2.root"
17-Sep-2025 02:45:11 CEST Closed input file "prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 229.113 362.044 577.134 332.675 99.5328 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00160484 0.00749814 0.0148397 0.00638117 0.00432205 10
reco:wclsdatahd:WireCellToolkit 65.7367 69.7979 74.3265 69.4327 3.49127 10
reco:gaushit:GausHitFinder 1.46659 1.91624 2.80126 1.78641 0.423582 10
reco:nhitsfilter:NumberOfHitsFilter 0.000558261 0.000926261 0.00149158 0.000728315 0.000352362 10
reco:reco3d:SpacePointSolver 0.923338 1.98276 3.13032 2.00999 0.73816 10
reco:hitpdune:DisambigFromSpacePoints 1.20687 2.4198 4.12553 2.24867 0.867712 10
reco:pandora:StandardPandora 26.6839 64.093 99.975 58.6178 22.3687 10
reco:pandoraWriter:StandardPandora 0.446335 0.586578 0.835288 0.533947 0.125785 10
reco:pandoraTrack:LArPandoraTrackCreation 3.39012 6.48226 13.4604 5.73676 2.79216 10
reco:pandoraShower:LArPandoraShowerCreation 3.63651 7.99368 17.2437 7.10428 3.92146 10
reco:pandoracalo:Calorimetry 1.53178 2.94044 6.37774 2.57004 1.33535 10
reco:pandoracalonosce:Calorimetry 1.52611 2.87653 6.47003 2.51353 1.39035 10
reco:pandoraShowercalo:ShowerCalorimetry 3.20359 7.42646 17.7225 6.09021 4.24293 10
reco:pandoraShowercalonosce:ShowerCalorimetry 2.85409 7.12745 17.1171 5.8486 4.15048 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.106811 0.14631 0.196727 0.137093 0.0290026 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.109365 0.138142 0.183196 0.13718 0.0207527 10
reco:emtrkmichelid:EmTrackMichelId 49.0426 64.2706 87.2331 60.9564 12.7272 10
reco:opdec:Deconvolution 1.34446 1.58997 2.45224 1.53045 0.307626 10
reco:ophitspe:OpHitFinderDeco 0.378782 0.432414 0.488216 0.447199 0.032356 10
reco:opflash:OpFlashFinder 0.0905772 0.136776 0.2515 0.130717 0.0437743 10
reco:opslicer:OpSlicer 0.409403 0.512021 0.770962 0.47467 0.103279 10
reco:crtreco:TwoCRTMatchingProducer 0.0134372 0.022195 0.0295643 0.0236868 0.00484278 10
reco:crttag:SingleCRTMatchingProducer 0.0109626 0.0174023 0.0258361 0.0168674 0.00424726 10
reco:pandora2Track:LArPandoraTrackCreation 10.723 21.0049 43.1876 18.6141 9.39419 10
reco:pandora2calo:ShowerCalorimetry 2.87753 6.87969 16.1353 5.60178 3.8553 10
reco:pandora2calonosce:ShowerCalorimetry 3.0291 6.70313 15.7677 5.70685 3.70341 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.11307 0.14345 0.205068 0.13866 0.0266093 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.111973 0.138578 0.199224 0.131207 0.0254652 10
reco:pandora2Shower:LArPandoraShowerCreation 4.33802 8.96417 19.0567 7.79447 4.33213 10
reco:pandora2Showercalo:ShowerCalorimetry 3.70526 8.92762 21.8698 7.37806 5.15184 10
reco:pandora2Showercalonosce:ShowerCalorimetry 3.88098 8.50266 20.5615 7.15951 4.81523 10
[art]:TriggerResults:TriggerResultInserter 4.1027e-05 6.45649e-05 0.000110016 5.39485e-05 2.37451e-05 10
end_path:out1:RootOutput 1.1779e-05 1.5758e-05 3.4405e-05 1.29205e-05 6.71998e-06 10
end_path:michelremoving:michelremoving 0.0996836 0.473573 0.864527 0.420796 0.243241 10
end_path:michelremovinggnocchi:michelremoving 0.0784046 0.439459 0.81935 0.384957 0.231705 10
end_path:calibanagnocchi:CalibAnaTree 30.1148 52.5168 81.8865 50.2282 15.819 10
end_path:out1:RootOutput(write) 3.49871 4.28141 6.03454 4.14586 0.743625 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7881.29 MB
Peak resident set size usage (VmHWM): 5481.49 MB
Details saved in: 'mem.db'
====================================================================================================
{'art_out': 'prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1_reco2.root', 'tfile_out': 'prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1_reco2_tfile.root', 'start_time': 1758066065.443889, 'end_time': 1758069914.8686438}
lsing
total 18G
-rw-r--r-- 1 fermilab users 6.2G Sep 17 01:41 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1.root
-rw-r--r-- 1 fermilab users 5.7G Sep 17 01:34 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim.root
-rw-r--r-- 1 fermilab users 5.4G Sep 17 02:45 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1_reco2.root
-rw-r--r-- 1 fermilab users 56M Sep 17 02:45 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1_reco2_tfile.root
-rw-r--r-- 1 fermilab users 52M Sep 17 02:38 Pandora_Events.pndr
-rw-r--r-- 1 fermilab users 220K Sep 17 02:45 mem.db
-rw-r--r-- 1 fermilab users 200K Sep 17 02:45 jobscript.log
-rw-r--r-- 1 fermilab users 40K Sep 17 02:45 time.db
-rw-r--r-- 1 fermilab users 14K Sep 17 02:45 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1_reco2.log
-rw-r--r-- 1 fermilab users 13K Sep 17 02:45 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1_reco2.err
-rw-r--r-- 1 fermilab users 13K Sep 17 01:34 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim.log
-rw-r--r-- 1 fermilab users 6.8K Sep 17 01:41 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1.err
-rw-r--r-- 1 fermilab users 4.5K Sep 17 01:34 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim.err
-rw-r--r-- 1 fermilab users 3.5K Sep 17 02:45 Pandora_Geometry.xml
-rw-r--r-- 1 fermilab users 1.4K Sep 17 01:41 prod_beam_p5GeV_protodunehd_1_40628_0_1758059494_detsim_reco1.log
-rw-r--r-- 1 fermilab users 519 Sep 17 01:34 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 fermilab users 519 Sep 17 01:41 reco_protodunehd_hist.root
-rw-r--r-- 1 fermilab users 109 Sep 16 23:51 all-input-dids.txt
-rw-r--r-- 1 fermilab users 0 Sep 17 01:47 ProtoDUNE.log
-rw-r--r-- 1 fermilab users 0 Sep 17 01:41 debugprod.log
disk usage
18G . 