Jobsub ID 43063.176@dunegpschedd01.fnal.gov
| Jobsub ID | 43063.176@dunegpschedd01.fnal.gov |
| Workflow ID | 2327 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:21:53 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce4 |
| Last heartbeat | 2025-09-08 15:52:48 |
| From worker node | Hostname | n196.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 13:42:15 |
| Input files | vd-protodune:np02vd_raw_run039324_2470_df-s02-d0_dw_0_20250907T201857.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7216s) |
| CPU time | 1h (4997s = 69%) |
| Max RSS bytes | 3168235520 (3021 MiB) |
| Outputting started | 2025-09-08 15:42:32 |
| Output files | |
| Finished | 2025-09-08 15:52:48 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
e
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 13.12 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.4 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.35 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.99 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.9 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.61 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.52 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.99 sec
[16:39:16.932] I [ timer ] Timer: WireCell::Aux::Resampler : 1.05 sec
[16:39:16.932] I [ timer ] Timer: WireCell::Aux::Resampler : 1.04 sec
[16:39:16.932] I [ timer ] Timer: WireCell::Aux::Resampler : 1.03 sec
[16:39:16.932] I [ timer ] Timer: WireCell::Aux::Resampler : 1.02 sec
[16:39:16.932] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[16:39:16.932] I [ timer ] Timer: WireCell::Gen::Retagger : 0.02 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:39:16.932] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:39:16.932] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:39:16.932] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:39:16.932] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:39:16.932] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:39:16.932] I [ timer ] Timer: Total node execution : 100.11999902687967 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 56252 traces tagged "gauss"
FrameSaver: q=8.72585e+06 n=1117928 tag=gauss
wclsFrameSaver: saving 67890 traces tagged "wiener"
FrameSaver: q=9.12979e+06 n=1071781 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1257 XUs and 1505 XVs -> 48 XUVs
C:0 T:1 181 XUs and 198 XVs -> 6 XUVs
C:0 T:2 597 XUs and 562 XVs -> 42 XUVs
C:0 T:3 548 XUs and 683 XVs -> 13 XUVs
C:0 T:4 1241 XUs and 1301 XVs -> 42 XUVs
C:0 T:5 687 XUs and 717 XVs -> 23 XUVs
C:0 T:6 627 XUs and 594 XVs -> 26 XUVs
C:0 T:7 2421 XUs and 2491 XVs -> 89 XUVs
C:0 T:8 6383 XUs and 7308 XVs -> 343 XUVs
C:0 T:9 2992 XUs and 3408 XVs -> 234 XUVs
C:0 T:10 2492 XUs and 2311 XVs -> 109 XUVs
C:0 T:11 2613 XUs and 3350 XVs -> 225 XUVs
C:0 T:12 1636 XUs and 2135 XVs -> 141 XUVs
C:0 T:13 3458 XUs and 2574 XVs -> 143 XUVs
C:0 T:14 1543 XUs and 1430 XVs -> 68 XUVs
C:0 T:15 3013 XUs and 2896 XVs -> 145 XUVs
1697 XUVs total
1418 collection wire objects
1697 potential space points
Neighbour search...
14597 tests to find 7560 neighbours
Iterating with no regularization...
Begin: 4.5826e+08
0 4.44286e+08
1 4.44087e+08
Now with regularization...
Begin: 4.3762e+08
0 4.376e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 16:41:19 BST Closed output file "np02vd_raw_run039324_2470_df-s02-d0_dw_0_20250907T201857_reco_stage1_20250908T154119_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 164.953 261.666 743.94 219.216 115.546 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.437e-05 7.38793e-05 0.000167705 6.8692e-05 2.14634e-05 27
produce:tpcrawdecoder:PDVDTPCReader 56.1591 74.4574 96.0273 77.0569 12.6541 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.059144 0.0648881 0.0844826 0.0602036 0.00659647 27
produce:pdvddaphne:DAPHNEReaderPDVD 7.69774 10.6512 13.8131 10.8585 1.63476 27
produce:ophit:OpHitFinder 0.0378449 0.0463752 0.0533676 0.0462907 0.00336554 27
produce:opflash:OpFlashFinderVerticalDrift 0.00651207 0.0137817 0.0180525 0.0141752 0.00273028 27
produce:wclsdatavd:WireCellToolkit 48.2118 64.0394 101.025 61.1394 10.2164 27
produce:gaushit:GausHitFinder 0.94065 1.55954 2.60685 1.46284 0.420157 27
produce:nhitsfilter:NumberOfHitsFilter 0.000284297 0.000407793 0.000574219 0.000373306 8.48325e-05 27
produce:reco3d:SpacePointSolver 7.09837 13.9725 27.9838 13.0061 4.84193 27
produce:hitpdune:DisambigFromSpacePoints 0.131325 0.277965 0.640738 0.248072 0.117101 27
produce:pandora:StandardPandora 27.3762 90.9087 547.868 53.2225 107.049 27
produce:pandoraTrack:LArPandoraTrackCreation 0.33238 1.3075 4.15399 0.871694 0.925267 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.0158844 0.0295349 0.0505952 0.0270075 0.00897478 27
[art]:TriggerResults:TriggerResultInserter 2.2339e-05 3.32849e-05 7.7974e-05 3.0393e-05 1.15457e-05 27
end_path:out1:RootOutput 3.644e-06 8.29185e-06 2.2076e-05 7.112e-06 3.73214e-06 27
end_path:out1:RootOutput(write) 3.83864 4.27233 6.10033 4.10634 0.521965 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5063.03 MB
Peak resident set size usage (VmHWM): 3168.24 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2470_df-s02-d0_dw_0_20250907T201857_reco_stage1_20250908T154119_keepup.root
\tHists: np02vd_raw_run039324_2470_df-s02-d0_dw_0_20250907T201857_reco_stage1_20250908T154120_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2470_df-s02-d0_dw_0_20250907T201857_reco_stage1_20250908T154119_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2470_df-s02-d0_dw_0_20250907T201857_reco_stage1_20250908T154119_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346080.0,
"core.end_time": 1757346080.0,
"core.events": [
1294162,
1294182,
1294202,
1294222,
1294242,
1294262,
1294282,
1294302,
1294322,
1294342,
1294362,
1294382,
1294402,
1294422,
1294442,
1294462,
1294482,
1294502,
1294522,
1294542,
1294562,
1294582,
1294602,
1294622,
1294642,
1294662,
1294682
],
"core.event_count": 27,
"core.first_event_number": 1294162,
"core.last_event_number": 1294682,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2470_df-s02-d0_dw_0_20250907T201857.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2470_df-s02-d0_dw_0_20250907T201857_reco_stage1_20250908T154120_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346080.0,
"core.end_time": 1757346080.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2470_df-s02-d0_dw_0_20250907T201857.hdf5"
}
]
} 