Jobsub ID 36361.53@dunegpschedd02.fnal.gov
| Jobsub ID | 36361.53@dunegpschedd02.fnal.gov |
| Workflow ID | 2329 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 12:39:56 |
| Site | UK_Oxford |
| Entry | DUNE_UK_SGrid_Oxford_arc01 |
| Last heartbeat | 2025-09-08 15:52:45 |
| From worker node | Hostname | t2wn065.physics.ox.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:42:23 |
| Input files | vd-protodune:np02vd_raw_run039324_2866_df-s05-d5_dw_0_20250908T034745.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (3627s) |
| CPU time | 52m (3168s = 87%) |
| Max RSS bytes | 3029647360 (2889 MiB) |
| Outputting started | 2025-09-08 15:42:50 |
| Output files | |
| Finished | 2025-09-08 15:52:45 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.31 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.72 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.6 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.4 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.8 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.78 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.7 sec
[16:40:02.518] I [ timer ] Timer: WireCell::Aux::Resampler : 0.46 sec
[16:40:02.518] I [ timer ] Timer: WireCell::Aux::Resampler : 0.36 sec
[16:40:02.518] I [ timer ] Timer: WireCell::Aux::Resampler : 0.3 sec
[16:40:02.518] I [ timer ] Timer: WireCell::Aux::Resampler : 0.28 sec
[16:40:02.518] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:02.518] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:02.518] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:40:02.518] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:02.518] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:40:02.518] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:40:02.518] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:40:02.518] I [ timer ] Timer: Total node execution : 49.150000320747495 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 35106 traces tagged "gauss"
FrameSaver: q=6.10816e+06 n=765903 tag=gauss
wclsFrameSaver: saving 44939 traces tagged "wiener"
FrameSaver: q=6.53624e+06 n=725916 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 483 XUs and 848 XVs -> 27 XUVs
C:0 T:1 142 XUs and 194 XVs -> 14 XUVs
C:0 T:2 333 XUs and 459 XVs -> 32 XUVs
C:0 T:3 997 XUs and 1698 XVs -> 197 XUVs
C:0 T:4 133 XUs and 262 XVs -> 4 XUVs
C:0 T:5 202 XUs and 190 XVs -> 4 XUVs
C:0 T:6 1050 XUs and 1507 XVs -> 251 XUVs
C:0 T:7 245 XUs and 332 XVs -> 33 XUVs
C:0 T:8 5112 XUs and 6927 XVs -> 767 XUVs
C:0 T:9 7588 XUs and 9485 XVs -> 1010 XUVs
C:0 T:10 14723 XUs and 21618 XVs -> 3915 XUVs
C:0 T:11 9612 XUs and 11906 XVs -> 1958 XUVs
C:0 T:12 3797 XUs and 5436 XVs -> 1010 XUVs
C:0 T:13 3893 XUs and 3701 XVs -> 538 XUVs
C:0 T:14 2065 XUs and 2899 XVs -> 700 XUVs
C:0 T:15 769 XUs and 764 XVs -> 46 XUVs
10506 XUVs total
1698 collection wire objects
10506 potential space points
Neighbour search...
1390482 tests to find 591548 neighbours
Iterating with no regularization...
Begin: 3.31072e+08
0 2.82733e+08
1 2.79374e+08
2 2.78852e+08
3 2.78695e+08
Now with regularization...
Begin: 2.71496e+08
0 2.71357e+08
PcaShowerParticleBuildingAlgorithm::OpeningAngle - principal eigenvalue less than or equal to 0.
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 16:41:23 BST Closed output file "np02vd_raw_run039324_2866_df-s05-d5_dw_0_20250908T034745_reco_stage1_20250908T154123_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 97.9597 145.154 244.393 133.102 38.846 24
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 7.3427e-05 0.000164779 0.000501548 0.000158246 8.42136e-05 24
produce:tpcrawdecoder:PDVDTPCReader 12.2052 15.6965 19.7206 15.0374 2.57231 24
produce:triggerrawdecoder:PDVDTriggerReader4 0.0369536 0.0385435 0.0568385 0.0374509 0.00394089 24
produce:pdvddaphne:DAPHNEReaderPDVD 3.28793 3.75311 4.68395 3.63763 0.36949 24
produce:ophit:OpHitFinder 0.0352867 0.0592309 0.081002 0.0577951 0.0108839 24
produce:opflash:OpFlashFinderVerticalDrift 0.00671273 0.0147358 0.0228537 0.0144768 0.00374604 24
produce:wclsdatavd:WireCellToolkit 43.6141 57.5021 82.434 50.5476 13.5348 24
produce:gaushit:GausHitFinder 0.74449 1.35272 2.41497 1.3626 0.401285 24
produce:nhitsfilter:NumberOfHitsFilter 0.000266118 0.000586865 0.00181424 0.000455476 0.00032216 24
produce:reco3d:SpacePointSolver 7.68609 13.2066 24.6662 12.5249 4.48359 24
produce:hitpdune:DisambigFromSpacePoints 0.0721349 0.202763 0.429316 0.188015 0.0911803 24
produce:pandora:StandardPandora 16.5408 47.6571 115.877 40.1996 24.0684 24
produce:pandoraTrack:LArPandoraTrackCreation 0.33645 0.762321 1.69868 0.63423 0.35037 24
produce:pandoraGnocalo:GnocchiCalorimetry 0.0161711 0.0310884 0.0569745 0.0306254 0.0104119 24
[art]:TriggerResults:TriggerResultInserter 2.5788e-05 4.838e-05 0.000186719 4.10815e-05 3.0678e-05 24
end_path:out1:RootOutput 5.39e-06 1.36056e-05 7.2906e-05 1.1477e-05 1.26155e-05 24
end_path:out1:RootOutput(write) 3.61225 4.81855 7.31806 4.31976 1.08115 24
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5038.23 MB
Peak resident set size usage (VmHWM): 3029.65 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2866_df-s05-d5_dw_0_20250908T034745_reco_stage1_20250908T154123_keepup.root
\tHists: np02vd_raw_run039324_2866_df-s05-d5_dw_0_20250908T034745_reco_stage1_20250908T154123_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2866_df-s05-d5_dw_0_20250908T034745_reco_stage1_20250908T154123_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2866_df-s05-d5_dw_0_20250908T034745_reco_stage1_20250908T154123_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346084.0,
"core.end_time": 1757346084.0,
"core.events": [
1503399,
1503419,
1503439,
1503459,
1503479,
1503499,
1503519,
1503539,
1503559,
1503579,
1503599,
1503619,
1503639,
1503659,
1503679,
1503699,
1503719,
1503739,
1503759,
1503779,
1503799,
1503819,
1503839,
1503859
],
"core.event_count": 24,
"core.first_event_number": 1503399,
"core.last_event_number": 1503859,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2866_df-s05-d5_dw_0_20250908T034745.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2866_df-s05-d5_dw_0_20250908T034745_reco_stage1_20250908T154123_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346084.0,
"core.end_time": 1757346084.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2866_df-s05-d5_dw_0_20250908T034745.hdf5"
}
]
} 