Jobsub ID 36377.50@dunegpschedd02.fnal.gov
| Jobsub ID | 36377.50@dunegpschedd02.fnal.gov |
| Workflow ID | 2327 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 13:33:57 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce505 |
| Last heartbeat | 2025-09-08 15:52:31 |
| From worker node | Hostname | b9p02p3820.cern.ch |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:57:19 |
| Input files | vd-protodune:np02vd_raw_run039324_2538_df-s03-d1_dw_0_20250907T213137.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 52m (3166s) |
| CPU time | 46m (2780s = 87%) |
| Max RSS bytes | 3090214912 (2947 MiB) |
| Outputting started | 2025-09-08 15:50:06 |
| Output files | |
| Finished | 2025-09-08 15:52:31 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
::SigProc::OmnibusSigProc : 6.06 sec
[17:43:28.647] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.7 sec
[17:43:28.647] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.7 sec
[17:43:28.647] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.07 sec
[17:43:28.647] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.98 sec
[17:43:28.647] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.69 sec
[17:43:28.648] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.27 sec
[17:43:28.648] I [ timer ] Timer: WireCell::Aux::Resampler : 0.39 sec
[17:43:28.648] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[17:43:28.648] I [ timer ] Timer: WireCell::Aux::Resampler : 0.24 sec
[17:43:28.648] I [ timer ] Timer: WireCell::Aux::Resampler : 0.24 sec
[17:43:28.648] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[17:43:28.648] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[17:43:28.648] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[17:43:28.648] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[17:43:28.648] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:28.648] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:28.648] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:28.648] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:28.648] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:43:28.648] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:28.648] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:28.648] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:43:28.648] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:43:28.648] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:43:28.648] I [ timer ] Timer: Total node execution : 45.89999953471124 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 54630 traces tagged "gauss"
FrameSaver: q=1.06227e+07 n=1297320 tag=gauss
wclsFrameSaver: saving 69673 traces tagged "wiener"
FrameSaver: q=1.15094e+07 n=1244592 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 786 XUs and 807 XVs -> 31 XUVs
C:0 T:1 1304 XUs and 1228 XVs -> 42 XUVs
C:0 T:2 667 XUs and 625 XVs -> 26 XUVs
C:0 T:3 454 XUs and 955 XVs -> 24 XUVs
C:0 T:4 560 XUs and 648 XVs -> 27 XUVs
C:0 T:5 457 XUs and 545 XVs -> 16 XUVs
C:0 T:6 812 XUs and 823 XVs -> 22 XUVs
C:0 T:7 1438 XUs and 1607 XVs -> 19 XUVs
C:0 T:8 28386 XUs and 55178 XVs -> 6769 XUVs
C:0 T:9 8343 XUs and 14633 XVs -> 864 XUVs
C:0 T:10 11564 XUs and 30026 XVs -> 3949 XUVs
C:0 T:11 11248 XUs and 17711 XVs -> 2957 XUVs
C:0 T:12 6510 XUs and 10336 XVs -> 1223 XUVs
C:0 T:13 9832 XUs and 18185 XVs -> 2269 XUVs
C:0 T:14 13185 XUs and 19974 XVs -> 4439 XUVs
C:0 T:15 10692 XUs and 17377 XVs -> 3128 XUVs
25805 XUVs total
3191 collection wire objects
25805 potential space points
Neighbour search...
3335817 tests to find 1417042 neighbours
Iterating with no regularization...
Begin: 6.24734e+08
0 5.02682e+08
1 4.93645e+08
2 4.92158e+08
3 4.91661e+08
4 4.91432e+08
Now with regularization...
Begin: 4.71576e+08
0 4.71283e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 17:45:38 CEST Closed output file "np02vd_raw_run039324_2538_df-s03-d1_dw_0_20250907T213137_reco_stage1_20250908T154538_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 55.9259 110.067 218.759 96.2389 41.3227 26
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 4.5231e-05 8.03966e-05 0.000163611 7.02355e-05 3.06823e-05 26
produce:tpcrawdecoder:PDVDTPCReader 3.65822 5.66766 15.2674 4.09823 3.21803 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.000739994 0.00226654 0.0213238 0.00108443 0.00432972 26
produce:pdvddaphne:DAPHNEReaderPDVD 1.94568 2.43069 3.4081 2.32641 0.334651 26
produce:ophit:OpHitFinder 0.0295436 0.0404834 0.0509289 0.039545 0.00551339 26
produce:opflash:OpFlashFinderVerticalDrift 0.00347197 0.0101321 0.0218545 0.00872172 0.00415652 26
produce:wclsdatavd:WireCellToolkit 33.3623 41.3252 60.7992 39.1184 7.44586 26
produce:gaushit:GausHitFinder 0.479549 0.992134 1.80577 0.916923 0.383212 26
produce:nhitsfilter:NumberOfHitsFilter 0.000136161 0.000238683 0.000609203 0.000209682 0.000102329 26
produce:reco3d:SpacePointSolver 3.44262 10.5843 19.6765 9.49334 4.45984 26
produce:hitpdune:DisambigFromSpacePoints 0.0413198 0.168337 0.492899 0.135119 0.10863 26
produce:pandora:StandardPandora 7.03739 44.7564 139.722 32.8436 32.4199 26
produce:pandoraTrack:LArPandoraTrackCreation 0.138762 0.509115 0.943199 0.474886 0.22559 26
produce:pandoraGnocalo:GnocchiCalorimetry 0.00770787 0.0205909 0.0351656 0.0206828 0.00651818 26
[art]:TriggerResults:TriggerResultInserter 1.688e-05 2.39237e-05 5.705e-05 2.2735e-05 7.82168e-06 26
end_path:out1:RootOutput 3.87e-06 6.69504e-06 2.706e-05 5.895e-06 4.22197e-06 26
end_path:out1:RootOutput(write) 2.92203 3.53385 5.35505 3.42947 0.564141 26
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5046.87 MB
Peak resident set size usage (VmHWM): 3090.21 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2538_df-s03-d1_dw_0_20250907T213137_reco_stage1_20250908T154538_keepup.root
\tHists: np02vd_raw_run039324_2538_df-s03-d1_dw_0_20250907T213137_reco_stage1_20250908T154538_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2538_df-s03-d1_dw_0_20250907T213137_reco_stage1_20250908T154538_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2538_df-s03-d1_dw_0_20250907T213137_reco_stage1_20250908T154538_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346338.0,
"core.end_time": 1757346338.0,
"core.events": [
1331367,
1331387,
1331407,
1331427,
1331447,
1331467,
1331487,
1331507,
1331527,
1331547,
1331567,
1331587,
1331607,
1331627,
1331647,
1331667,
1331687,
1331707,
1331727,
1331747,
1331767,
1331787,
1331807,
1331827,
1331847,
1331867
],
"core.event_count": 26,
"core.first_event_number": 1331367,
"core.last_event_number": 1331867,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2538_df-s03-d1_dw_0_20250907T213137.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2538_df-s03-d1_dw_0_20250907T213137_reco_stage1_20250908T154538_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346338.0,
"core.end_time": 1757346338.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2538_df-s03-d1_dw_0_20250907T213137.hdf5"
}
]
} 