Jobsub ID 36365.94@dunegpschedd02.fnal.gov
| Jobsub ID | 36365.94@dunegpschedd02.fnal.gov |
| Workflow ID | 2326 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 12:59:55 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce01 |
| Last heartbeat | 2025-09-08 15:52:34 |
| From worker node | Hostname | dune001-9022582.0-host-172-16-102-248.nubes.stfc.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:48:25 |
| Input files | vd-protodune:np02vd_raw_run039324_2278_df-s05-d5_dw_0_20250907T155418.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 59m (3574s) |
| CPU time | 51m (3061s = 85%) |
| Max RSS bytes | 3145723904 (2999 MiB) |
| Outputting started | 2025-09-08 15:47:59 |
| Output files | |
| Finished | 2025-09-08 15:52:34 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
[15:42:17.062] D [ glue ] <FrameFanin:nfsp> EOS at call=43 with 8
[15:42:17.062] D [ glue ] frame sink sees EOS
[15:42:17.062] D [ pgraph ] <Pgrapher:> graph execution complete
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.86 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.28 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.11 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.97 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.65 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.5 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.94 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.88 sec
[15:42:17.062] I [ timer ] Timer: WireCell::Aux::Resampler : 0.68 sec
[15:42:17.062] I [ timer ] Timer: WireCell::Aux::Resampler : 0.66 sec
[15:42:17.062] I [ timer ] Timer: WireCell::Aux::Resampler : 0.65 sec
[15:42:17.062] I [ timer ] Timer: WireCell::Aux::Resampler : 0.64 sec
[15:42:17.062] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:17.062] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[15:42:17.062] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:17.062] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[15:42:17.062] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[15:42:17.062] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[15:42:17.062] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[15:42:17.062] I [ timer ] Timer: Total node execution : 56.88000052794814 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 46859 traces tagged "gauss"
FrameSaver: q=7.45161e+06 n=941070 tag=gauss
wclsFrameSaver: saving 55638 traces tagged "wiener"
FrameSaver: q=7.78407e+06 n=913686 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 100 XUs and 73 XVs -> 3 XUVs
C:0 T:1 328 XUs and 305 XVs -> 8 XUVs
C:0 T:2 532 XUs and 511 XVs -> 16 XUVs
C:0 T:3 745 XUs and 744 XVs -> 28 XUVs
C:0 T:4 1546 XUs and 1460 XVs -> 53 XUVs
C:0 T:5 760 XUs and 655 XVs -> 32 XUVs
C:0 T:6 204 XUs and 225 XVs -> 11 XUVs
C:0 T:7 66 XUs and 92 XVs -> 2 XUVs
C:0 T:8 2796 XUs and 2915 XVs -> 138 XUVs
C:0 T:9 1000 XUs and 1055 XVs -> 60 XUVs
C:0 T:10 1386 XUs and 1693 XVs -> 75 XUVs
C:0 T:11 2271 XUs and 2579 XVs -> 149 XUVs
C:0 T:12 5053 XUs and 7748 XVs -> 336 XUVs
C:0 T:13 1336 XUs and 2178 XVs -> 87 XUVs
C:0 T:14 1088 XUs and 1466 XVs -> 73 XUVs
C:0 T:15 229 XUs and 271 XVs -> 13 XUVs
1084 XUVs total
924 collection wire objects
1084 potential space points
Neighbour search...
7362 tests to find 3044 neighbours
Iterating with no regularization...
Begin: 3.02881e+08
0 2.94841e+08
1 2.9479e+08
Now with regularization...
Begin: 2.90277e+08
0 2.90274e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 15:44:01 UTC Closed output file "np02vd_raw_run039324_2278_df-s05-d5_dw_0_20250907T155418_reco_stage1_20250908T154401_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 83.8591 148.747 253.718 144.716 45.1496 22
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.4651e-05 0.000253083 0.0031403 0.000101797 0.000633547 22
produce:tpcrawdecoder:PDVDTPCReader 11.836 15.1232 20.6222 14.6015 2.86511 22
produce:triggerrawdecoder:PDVDTriggerReader4 0.0362696 0.0375307 0.038704 0.0374755 0.00063712 22
produce:pdvddaphne:DAPHNEReaderPDVD 2.90192 3.67904 4.96828 3.65082 0.576887 22
produce:ophit:OpHitFinder 0.0286963 0.0449986 0.0662328 0.0447106 0.00782665 22
produce:opflash:OpFlashFinderVerticalDrift 0.00524376 0.0120833 0.0202639 0.0115086 0.00421967 22
produce:wclsdatavd:WireCellToolkit 33.143 55.1063 78.6837 51.4417 14.2387 22
produce:gaushit:GausHitFinder 0.536354 1.23854 2.19764 1.07738 0.427673 22
produce:nhitsfilter:NumberOfHitsFilter 0.000187893 0.000314914 0.000545667 0.000292563 8.94827e-05 22
produce:reco3d:SpacePointSolver 5.02931 12.546 21.8745 11.7892 3.9197 22
produce:hitpdune:DisambigFromSpacePoints 0.0694681 0.20436 0.407233 0.183963 0.0846926 22
produce:pandora:StandardPandora 10.418 54.8768 136.579 47.0803 31.254 22
produce:pandoraTrack:LArPandoraTrackCreation 0.328049 0.934527 1.66202 0.731345 0.471834 22
produce:pandoraGnocalo:GnocchiCalorimetry 0.0158593 0.0317383 0.0491776 0.0304934 0.0116553 22
[art]:TriggerResults:TriggerResultInserter 2.194e-05 4.38307e-05 0.000165192 3.67655e-05 2.94152e-05 22
end_path:out1:RootOutput 5.22e-06 1.18946e-05 5.4951e-05 1.05805e-05 9.65276e-06 22
end_path:out1:RootOutput(write) 3.08576 4.8691 7.28243 4.52996 1.51141 22
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5268.66 MB
Peak resident set size usage (VmHWM): 3145.72 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2278_df-s05-d5_dw_0_20250907T155418_reco_stage1_20250908T154401_keepup.root
\tHists: np02vd_raw_run039324_2278_df-s05-d5_dw_0_20250907T155418_reco_stage1_20250908T154401_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2278_df-s05-d5_dw_0_20250907T155418_reco_stage1_20250908T154401_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2278_df-s05-d5_dw_0_20250907T155418_reco_stage1_20250908T154401_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346242.0,
"core.end_time": 1757346242.0,
"core.events": [
1194019,
1194039,
1194059,
1194079,
1194099,
1194119,
1194139,
1194159,
1194179,
1194199,
1194219,
1194239,
1194259,
1194279,
1194299,
1194319,
1194339,
1194359,
1194379,
1194399,
1194419,
1194439
],
"core.event_count": 22,
"core.first_event_number": 1194019,
"core.last_event_number": 1194439,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2278_df-s05-d5_dw_0_20250907T155418.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2278_df-s05-d5_dw_0_20250907T155418_reco_stage1_20250908T154401_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346242.0,
"core.end_time": 1757346242.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2278_df-s05-d5_dw_0_20250907T155418.hdf5"
}
]
} 