Jobsub ID 36371.188@dunegpschedd02.fnal.gov
| Jobsub ID | 36371.188@dunegpschedd02.fnal.gov |
| Workflow ID | 2327 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 13:23:57 |
| Site | IT_CNAF |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 |
| Last heartbeat | 2025-09-08 15:52:37 |
| From worker node | Hostname | cn-leo-040.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) Platinum 8480+ |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:51:20 |
| Input files | vd-protodune:np02vd_raw_run039324_2536_df-s02-d1_dw_0_20250907T212702.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 59m (3556s) |
| CPU time | 48m (2882s = 81%) |
| Max RSS bytes | 3066769408 (2924 MiB) |
| Outputting started | 2025-09-08 15:50:37 |
| Output files | |
| Finished | 2025-09-08 15:52:37 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
[ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.23 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.9 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.75 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.67 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.64 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.63 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.57 sec
[17:45:18.677] I [ timer ] Timer: WireCell::Aux::Resampler : 0.23 sec
[17:45:18.677] I [ timer ] Timer: WireCell::Aux::Resampler : 0.21 sec
[17:45:18.677] I [ timer ] Timer: WireCell::Aux::Resampler : 0.2 sec
[17:45:18.677] I [ timer ] Timer: WireCell::Aux::Resampler : 0.2 sec
[17:45:18.677] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:45:18.677] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:45:18.677] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:45:18.677] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[17:45:18.677] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:45:18.677] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:45:18.677] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:45:18.677] I [ timer ] Timer: Total node execution : 48.86000049859285 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 35190 traces tagged "gauss"
FrameSaver: q=6.34406e+06 n=809476 tag=gauss
wclsFrameSaver: saving 43109 traces tagged "wiener"
FrameSaver: q=6.77741e+06 n=783116 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 84 XUs and 107 XVs -> 2 XUVs
C:0 T:1 179 XUs and 200 XVs -> 5 XUVs
C:0 T:2 811 XUs and 982 XVs -> 39 XUVs
C:0 T:3 339 XUs and 351 XVs -> 22 XUVs
C:0 T:4 363 XUs and 557 XVs -> 26 XUVs
C:0 T:5 222 XUs and 349 XVs -> 13 XUVs
C:0 T:6 1337 XUs and 1411 XVs -> 85 XUVs
C:0 T:7 370 XUs and 512 XVs -> 16 XUVs
C:0 T:8 31332 XUs and 53441 XVs -> 13873 XUVs
C:0 T:9 7057 XUs and 7595 XVs -> 401 XUVs
C:0 T:10 1459 XUs and 2451 XVs -> 507 XUVs
C:0 T:11 1019 XUs and 1612 XVs -> 72 XUVs
C:0 T:12 5963 XUs and 6543 XVs -> 475 XUVs
C:0 T:13 2105 XUs and 1909 XVs -> 88 XUVs
C:0 T:14 1105 XUs and 1622 XVs -> 54 XUVs
C:0 T:15 2977 XUs and 2085 XVs -> 351 XUVs
16029 XUVs total
1479 collection wire objects
16029 potential space points
Neighbour search...
3362511 tests to find 1323904 neighbours
Iterating with no regularization...
Begin: 4.43137e+08
0 3.81183e+08
1 3.78323e+08
2 3.77944e+08
3 3.77798e+08
Now with regularization...
Begin: 3.70618e+08
0 3.70467e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 17:46:13 CEST Closed output file "np02vd_raw_run039324_2536_df-s02-d1_dw_0_20250907T212702_reco_stage1_20250908T154613_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 85.8435 120.67 180.231 117.68 22.1765 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.551e-05 7.96126e-05 0.000146024 7.6382e-05 1.53643e-05 27
produce:tpcrawdecoder:PDVDTPCReader 9.38135 15.7364 33.5522 13.538 6.54925 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.0208541 0.0219778 0.0313844 0.0216498 0.00189943 27
produce:pdvddaphne:DAPHNEReaderPDVD 2.15196 3.63194 17.2464 2.76056 3.14613 27
produce:ophit:OpHitFinder 0.0234094 0.02975 0.0330593 0.030175 0.00239291 27
produce:opflash:OpFlashFinderVerticalDrift 0.00550871 0.00923281 0.0122659 0.00951018 0.00159815 27
produce:wclsdatavd:WireCellToolkit 45.7298 51.4631 70.1548 49.4217 5.57916 27
produce:gaushit:GausHitFinder 0.499376 0.818602 1.13295 0.821579 0.151768 27
produce:nhitsfilter:NumberOfHitsFilter 0.000182331 0.000301171 0.000412277 0.000305054 6.07587e-05 27
produce:reco3d:SpacePointSolver 5.57037 10.122 15.5641 9.44869 2.44966 27
produce:hitpdune:DisambigFromSpacePoints 0.0687313 0.125723 0.195461 0.121436 0.0378616 27
produce:pandora:StandardPandora 11.8652 35.2155 66.2205 29.2153 15.7669 27
produce:pandoraTrack:LArPandoraTrackCreation 0.23511 0.59642 1.01202 0.563723 0.190286 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.00995937 0.0167338 0.022646 0.0171857 0.0030341 27
[art]:TriggerResults:TriggerResultInserter 2.2248e-05 2.9004e-05 6.4814e-05 2.6984e-05 7.84488e-06 27
end_path:out1:RootOutput 4.83e-06 6.42793e-06 2.0339e-05 5.894e-06 2.76277e-06 27
end_path:out1:RootOutput(write) 2.48441 2.83881 4.05837 2.76607 0.31989 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4971.41 MB
Peak resident set size usage (VmHWM): 3066.77 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2536_df-s02-d1_dw_0_20250907T212702_reco_stage1_20250908T154613_keepup.root
\tHists: np02vd_raw_run039324_2536_df-s02-d1_dw_0_20250907T212702_reco_stage1_20250908T154613_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2536_df-s02-d1_dw_0_20250907T212702_reco_stage1_20250908T154613_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2536_df-s02-d1_dw_0_20250907T212702_reco_stage1_20250908T154613_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346373.0,
"core.end_time": 1757346373.0,
"core.events": [
1328883,
1328903,
1328923,
1328943,
1328963,
1328983,
1329003,
1329023,
1329043,
1329063,
1329083,
1329103,
1329123,
1329143,
1329163,
1329183,
1329203,
1329223,
1329243,
1329263,
1329283,
1329303,
1329323,
1329343,
1329363,
1329383,
1329403
],
"core.event_count": 27,
"core.first_event_number": 1328883,
"core.last_event_number": 1329403,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2536_df-s02-d1_dw_0_20250907T212702.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2536_df-s02-d1_dw_0_20250907T212702_reco_stage1_20250908T154613_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346373.0,
"core.end_time": 1757346373.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2536_df-s02-d1_dw_0_20250907T212702.hdf5"
}
]
} 