Jobsub ID 238780.1@dunegpschedd01.fnal.gov
| Jobsub ID | 238780.1@dunegpschedd01.fnal.gov |
| Workflow ID | 9368 |
| Stage ID | 1 |
| User name | pmelas@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 8388608000 (8000 MiB) |
| Wall seconds limit | 86400 (24 hours) |
| Submitted time | 2025-10-27 06:10:32 |
| Site | US_FNAL-FermiGrid |
| Entry | FNAL_GPGrid_ce03_mcore_op_duneonly |
| Last heartbeat | 2025-10-27 08:47:41 |
| From worker node | Hostname | dunegli-7381652-0-fnpc23042.fnal.gov |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 10485760000 (10000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-10-27 06:11:39 |
| Input files | monte-carlo-009368-000001
|
| Jobscript | Exit code | 0 |
| Real time | 2h (8957s) |
| CPU time | 2h (8887s = 99%) |
| Max RSS bytes | 4484009984 (4276 MiB) |
| Outputting started | 2025-10-27 08:40:56 |
| Output files | |
| Finished | 2025-10-27 08:47:41 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 145002 traces, 0 summary
Retagger: tagging trace set: gauss with 113553 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 113553 traces tagged "gauss"
FrameSaver: q=4.09972e+06 n=2113377 tag=gauss
wclsFrameSaver: saving 145002 traces tagged "wiener"
FrameSaver: q=4.3578e+06 n=1903614 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 45738 XUs and 45821 XVs -> 889 XUVs
C:0 T:1 123010 XUs and 140209 XVs -> 12357 XUVs
C:0 T:2 137016 XUs and 187248 XVs -> 10472 XUVs
C:0 T:3 46442 XUs and 65339 XVs -> 313 XUVs
C:0 T:4 8304 XUs and 10453 XVs -> 98 XUVs
C:0 T:5 40627 XUs and 48061 XVs -> 9600 XUVs
C:0 T:6 154188 XUs and 162151 XVs -> 7236 XUVs
C:0 T:7 61800 XUs and 56962 XVs -> 596 XUVs
C:0 T:8 12474 XUs and 10598 XVs -> 141 XUVs
C:0 T:9 121147 XUs and 178842 XVs -> 26400 XUVs
C:0 T:10 41848 XUs and 57162 XVs -> 9074 XUVs
C:0 T:11 76 XUs and 53 XVs -> 13 XUVs
77189 XUVs total
15728 collection wire objects
71366 potential space points
Neighbour search...
16683978 tests to find 6712788 neighbours
Iterating with no regularization...
Begin: 1.6833e+08
0 1.31518e+08
1 1.28357e+08
2 1.27911e+08
3 1.27794e+08
Now with regularization...
Begin: 1.08529e+08
0 1.08216e+08
1 1.08123e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410141658
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410141658
fLifetime: 30000 [us]
27-Oct-2025 08:40:55 UTC Closed output file "reco_stage1.root"
27-Oct-2025 08:40:55 UTC Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob: # events skipped: 0
====================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================================
Full event 175.926 274.399 365.924 277.194 60.349 10
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000715841 0.00321527 0.00539393 0.00324624 0.00140786 10
reco:ophit:OpHitFinder 0.211683 0.240132 0.295542 0.236456 0.0253362 10
reco:opflash:OpFlashFinder 0.140942 0.178574 0.257863 0.169796 0.0329879 10
reco:opslicer:OpSlicer 0.43528 0.514457 0.701906 0.489574 0.0910875 10
reco:caldata:DataPrepModule 3.378 3.5241 3.95376 3.50246 0.155163 10
reco:wclsdatasp:WireCellToolkit 22.6868 24.8676 32.281 24.0427 2.75682 10
reco:gaushit:GausHitFinder 0.691491 0.954328 1.25783 0.952351 0.158151 10
reco:nhitsfilter:NumberOfHitsFilter 0.000311027 0.00044702 0.000731952 0.000396693 0.000131008 10
reco:reco3d:SpacePointSolver 5.48094 15.0747 21.9267 15.6755 5.28942 10
reco:hitpdune:DisambigFromSpacePoints 0.808636 1.51882 2.3606 1.56619 0.439416 10
reco:pandora:StandardPandora 13.7817 34.9348 60.4278 34.0075 14.8063 10
reco:pandoraWriter:StandardPandora 0.229473 0.327651 0.441235 0.322966 0.0730815 10
reco:pandoraTrack:LArPandoraTrackCreation 1.51528 3.38356 6.18859 3.02049 1.23579 10
reco:pandoraShower:LArPandoraShowerCreation 1.5858 3.88087 6.57514 3.64863 1.5449 10
reco:pandoracalo:Calorimetry 0.940696 1.82395 2.96364 1.75682 0.525629 10
reco:pandoracalonosce:Calorimetry 0.854308 1.7066 2.83721 1.64961 0.509669 10
reco:pandorapid:Chi2ParticleID 0.00267812 0.00400299 0.00529522 0.00380507 0.00071646 10
reco:pandoracalinoxyzt:CalibrationdEdXPDSP 0.0279239 0.0372277 0.0455785 0.0371847 0.00615249 10
reco:pandoracalipidnoxyzt:Chi2ParticleID 0.00252674 0.00372956 0.00468204 0.00363406 0.000593726 10
reco:pandoraShowercalo:ShowerCalorimetry 1.80512 4.33537 6.97438 4.08506 1.61481 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.76394 4.16178 6.51812 3.97439 1.45983 10
reco:emtrkmichelid:EmTrackMichelId 99.4559 134.09 169.323 134.495 20.9398 10
reco:crttag:SingleCRTMatchingProducer 0.0078231 0.0105019 0.0154761 0.00947431 0.00277608 10
reco:crtreco:TwoCRTMatchingProducer 0.00426366 0.00603344 0.00850519 0.00592342 0.001186 10
reco:anodepiercerst0:T0RecoAnodePiercers 0.556923 0.778388 1.00957 0.761712 0.141217 10
reco:pandora2Track:LArPandoraTrackCreation 4.96625 11.7794 17.8436 11.6256 4.18217 10
reco:pandora2calo:Calorimetry 2.74689 5.84765 8.75023 5.92895 1.85321 10
reco:pandora2calonosce:Calorimetry 2.77535 5.41197 7.75666 5.44747 1.55903 10
reco:pandora2pid:Chi2ParticleID 0.00664188 0.0100089 0.0133699 0.0100531 0.00177101 10
reco:pandora2calinoxyzt:CalibrationdEdXPDSP 0.0361027 0.0473479 0.061072 0.0459366 0.00820659 10
reco:pandora2calipidnoxyzt:Chi2ParticleID 0.00661537 0.00975127 0.0127395 0.00970077 0.00168688 10
reco:pandora2Shower:LArPandoraShowerCreation 1.97541 3.59216 5.38165 3.55309 1.04401 10
reco:pandora2Showercalo:ShowerCalorimetry 2.60046 4.68419 6.60453 4.6831 1.22006 10
reco:pandora2Showercalonosce:ShowerCalorimetry 2.40507 4.26473 6.14185 4.27745 1.10355 10
[art]:TriggerResults:TriggerResultInserter 3.4745e-05 5.02697e-05 0.000106561 4.3011e-05 1.97379e-05 10
end_path:out1:RootOutput 7.364e-06 8.9168e-06 1.2223e-05 8.8465e-06 1.31756e-06 10
end_path:out1:RootOutput(write) 1.68452 2.30137 2.93779 2.26172 0.392515 10
====================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5295.22 MB
Peak resident set size usage (VmHWM): 4484.01 MB
Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor: Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor: digitReader: 0.05 sec/event
ToolBasedRawDigitPrepService:dtor: adcSampleFiller: 0.15 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleAdcToKe: 0.03 sec/event
ToolBasedRawDigitPrepService:dtor: pdspTailPedRemovalZKe: 2.20 sec/event
ToolBasedRawDigitPrepService:dtor: adcKeepAllSignalFinder: 0.08 sec/event
ToolBasedRawDigitPrepService:dtor: adcScaleKeToAdc: 0.03 sec/event
ToolBasedRawDigitPrepService:dtor: pdsp_RemoveBadChannels: 0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11713224
-rw-r--r-- 1 dunegli fnalgrid 2689101033 Oct 27 07:53 detsim_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 2549303633 Oct 27 07:52 detsim_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 2445110509 Oct 27 08:40 reco_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 2152319977 Oct 27 06:25 g4_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 2066091409 Oct 27 06:15 g4_stage1.root
-rw-r--r-- 1 dunegli fnalgrid 91804480 Oct 27 08:37 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid 243220 Oct 27 08:40 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid 217088 Oct 27 08:40 mem.db
-rw-r--r-- 1 dunegli fnalgrid 62376 Oct 27 08:40 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid 36864 Oct 27 08:40 time.db
-rw-r--r-- 1 dunegli fnalgrid 519 Oct 27 07:53 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid 519 Oct 27 06:25 g4_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid 519 Oct 27 08:40 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dunegli fnalgrid 52 Oct 27 06:11 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid 7 Oct 27 08:40 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid 0 Oct 27 07:58 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid 0 Oct 27 07:53 debugprod.log
