Jobsub ID 36362.14@dunegpschedd02.fnal.gov
| Jobsub ID | 36362.14@dunegpschedd02.fnal.gov |
| Workflow ID | 2330 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 12:47:59 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce05 |
| Last heartbeat | 2025-09-08 15:52:29 |
| From worker node | Hostname | dune001-7674890.0-lcg2721.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 9654 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:43:52 |
| Input files | vd-protodune:np02vd_raw_run039327_0000_df-s04-d0_dw_0_20250908T102954.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (3761s) |
| CPU time | 50m (3034s = 80%) |
| Max RSS bytes | 2667651072 (2544 MiB) |
| Outputting started | 2025-09-08 15:46:34 |
| Output files | |
| Finished | 2025-09-08 15:52:29 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
:42:00.356] D [ aux ] <Resampler:resmp0> EOS at call=37
[15:42:00.356] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=37 anode=0
[15:42:00.356] D [ glue ] <FrameFanin:nfsp> EOS at call=37 with 8
[15:42:00.356] D [ glue ] frame sink sees EOS
[15:42:00.356] D [ pgraph ] <Pgrapher:> graph execution complete
[15:42:00.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.82 sec
[15:42:00.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.82 sec
[15:42:00.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.25 sec
[15:42:00.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.03 sec
[15:42:00.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.84 sec
[15:42:00.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.55 sec
[15:42:00.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.27 sec
[15:42:00.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.71 sec
[15:42:00.356] I [ timer ] Timer: WireCell::Aux::Resampler : 0.95 sec
[15:42:00.356] I [ timer ] Timer: WireCell::Aux::Resampler : 0.66 sec
[15:42:00.356] I [ timer ] Timer: WireCell::Aux::Resampler : 0.61 sec
[15:42:00.357] I [ timer ] Timer: WireCell::Aux::Resampler : 0.55 sec
[15:42:00.357] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[15:42:00.357] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:42:00.357] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:42:00.357] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:42:00.357] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:00.357] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:00.357] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[15:42:00.357] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:00.357] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:00.357] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[15:42:00.357] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:42:00.357] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[15:42:00.357] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[15:42:00.357] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[15:42:00.357] I [ timer ] Timer: Total node execution : 59.11000047810376 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 45582 traces tagged "gauss"
FrameSaver: q=6.96305e+06 n=935916 tag=gauss
wclsFrameSaver: saving 54904 traces tagged "wiener"
FrameSaver: q=7.33284e+06 n=903432 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 851 XUs and 895 XVs -> 43 XUVs
C:0 T:1 695 XUs and 630 XVs -> 40 XUVs
C:0 T:2 629 XUs and 484 XVs -> 29 XUVs
C:0 T:3 1634 XUs and 1676 XVs -> 176 XUVs
C:0 T:4 473 XUs and 444 XVs -> 16 XUVs
C:0 T:5 601 XUs and 485 XVs -> 45 XUVs
C:0 T:6 1232 XUs and 1140 XVs -> 106 XUVs
C:0 T:7 462 XUs and 516 XVs -> 28 XUVs
C:0 T:8 1742 XUs and 2685 XVs -> 104 XUVs
C:0 T:9 2510 XUs and 3952 XVs -> 171 XUVs
C:0 T:10 1057 XUs and 1141 XVs -> 77 XUVs
C:0 T:11 4033 XUs and 5200 XVs -> 243 XUVs
C:0 T:12 613 XUs and 861 XVs -> 51 XUVs
C:0 T:13 1529 XUs and 1776 XVs -> 153 XUVs
C:0 T:14 122 XUs and 147 XVs -> 10 XUVs
C:0 T:15 954 XUs and 641 XVs -> 54 XUVs
1346 XUVs total
982 collection wire objects
1346 potential space points
Neighbour search...
14188 tests to find 8216 neighbours
Iterating with no regularization...
Begin: 3.71069e+08
0 3.61871e+08
1 3.61657e+08
Now with regularization...
Begin: 3.54504e+08
0 3.54487e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 15:42:51 UTC Closed output file "np02vd_raw_run039327_0000_df-s04-d0_dw_0_20250908T102954_reco_stage1_20250908T154251_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 131.019 183.173 254.626 187.281 34.8647 19
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.6956e-05 0.000168851 0.000493755 0.000153001 8.54108e-05 19
produce:tpcrawdecoder:PDVDTPCReader 20.7128 27.7891 69.9277 21.7057 12.6102 19
produce:triggerrawdecoder:PDVDTriggerReader4 0.691418 0.843191 2.47963 0.728582 0.395616 19
produce:pdvddaphne:DAPHNEReaderPDVD 0.000359222 0.000451473 0.000880128 0.000422588 0.000108823 19
produce:ophit:OpHitFinder 0.000220893 0.000715562 0.00699879 0.000338441 0.00149007 19
produce:opflash:OpFlashFinderVerticalDrift 6.4237e-05 0.000120473 0.00039763 9.967e-05 7.03558e-05 19
produce:wclsdatavd:WireCellToolkit 54.6503 74.012 92.5443 72.42 10.5907 19
produce:gaushit:GausHitFinder 0.868339 1.51855 2.29611 1.56296 0.332067 19
produce:nhitsfilter:NumberOfHitsFilter 0.000329317 0.00112149 0.00248328 0.00107347 0.000503711 19
produce:reco3d:SpacePointSolver 9.42099 15.4403 21.1486 14.3092 3.26216 19
produce:hitpdune:DisambigFromSpacePoints 0.089763 0.225192 0.404824 0.222462 0.0780059 19
produce:pandora:StandardPandora 24.3643 56.1636 126.927 50.9881 28.7905 19
produce:pandoraTrack:LArPandoraTrackCreation 0.565045 2.16511 6.45256 1.94014 1.39925 19
produce:pandoraGnocalo:GnocchiCalorimetry 0.0225495 0.0393705 0.0652439 0.036636 0.0129872 19
[art]:TriggerResults:TriggerResultInserter 3.0136e-05 6.34789e-05 0.000164258 5.4192e-05 3.39745e-05 19
end_path:out1:RootOutput 8.322e-06 1.53372e-05 3.977e-05 1.334e-05 7.05499e-06 19
end_path:out1:RootOutput(write) 3.87247 4.90204 5.89065 4.71865 0.643651 19
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4827.44 MB
Peak resident set size usage (VmHWM): 2667.65 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039327_0000_df-s04-d0_dw_0_20250908T102954_reco_stage1_20250908T154251_keepup.root
\tHists: np02vd_raw_run039327_0000_df-s04-d0_dw_0_20250908T102954_reco_stage1_20250908T154251_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039327_0000_df-s04-d0_dw_0_20250908T102954_reco_stage1_20250908T154251_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039327_0000_df-s04-d0_dw_0_20250908T102954_reco_stage1_20250908T154251_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346172.0,
"core.end_time": 1757346172.0,
"core.events": [
4,
8,
12,
16,
20,
24,
28,
32,
36,
40,
44,
48,
52,
56,
60,
64,
68,
72,
76
],
"core.event_count": 19,
"core.first_event_number": 4,
"core.last_event_number": 76,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39327
],
"core.runs_subruns": [
3932700001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039327_0000_df-s04-d0_dw_0_20250908T102954.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039327_0000_df-s04-d0_dw_0_20250908T102954_reco_stage1_20250908T154251_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346172.0,
"core.end_time": 1757346172.0,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39327
],
"core.runs_subruns": [
3932700001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039327_0000_df-s04-d0_dw_0_20250908T102954.hdf5"
}
]
} 