Jobsub ID 36335.168@dunegpschedd02.fnal.gov
| Jobsub ID | 36335.168@dunegpschedd02.fnal.gov |
| Workflow ID | 2327 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:47:55 |
| Site | UK_Lancaster |
| Entry | UBoone_UK_Lancaster_HEC_grendel_ce02 |
| Last heartbeat | 2025-09-08 15:52:44 |
| From worker node | Hostname | comp20-06 |
| cpuinfo | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:06:56 |
| Input files | vd-protodune:np02vd_raw_run039324_2517_df-s05-d1_dw_0_20250907T210910.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5716s) |
| CPU time | 1h (4002s = 70%) |
| Max RSS bytes | 3100971008 (2957 MiB) |
| Outputting started | 2025-09-08 15:42:13 |
| Output files | |
| Finished | 2025-09-08 15:52:44 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
D [ glue ] <FrameFanin:nfsp> EOS at call=51 with 8
[16:40:03.767] D [ glue ] frame sink sees EOS
[16:40:03.767] D [ pgraph ] <Pgrapher:> graph execution complete
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.48 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.45 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.23 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.99 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.55 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.47 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.34 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.23 sec
[16:40:03.768] I [ timer ] Timer: WireCell::Aux::Resampler : 0.33 sec
[16:40:03.768] I [ timer ] Timer: WireCell::Aux::Resampler : 0.32 sec
[16:40:03.768] I [ timer ] Timer: WireCell::Aux::Resampler : 0.3 sec
[16:40:03.768] I [ timer ] Timer: WireCell::Aux::Resampler : 0.3 sec
[16:40:03.768] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[16:40:03.768] I [ timer ] Timer: WireCell::Gen::Retagger : 0.02 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:03.768] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:03.768] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:40:03.768] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:03.768] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:40:03.768] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:40:03.768] I [ timer ] Timer: Total node execution : 56.08999979868531 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 36007 traces tagged "gauss"
FrameSaver: q=5.82803e+06 n=708777 tag=gauss
wclsFrameSaver: saving 45651 traces tagged "wiener"
FrameSaver: q=6.20392e+06 n=673486 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 862 XUs and 1112 XVs -> 57 XUVs
C:0 T:1 150 XUs and 327 XVs -> 13 XUVs
C:0 T:2 15 XUs and 17 XVs -> 1 XUVs
C:0 T:3 127 XUs and 169 XVs -> 16 XUVs
C:0 T:4 1770 XUs and 2621 XVs -> 130 XUVs
C:0 T:5 124 XUs and 284 XVs -> 6 XUVs
C:0 T:6 1974 XUs and 2467 XVs -> 90 XUVs
C:0 T:7 2480 XUs and 3216 XVs -> 188 XUVs
C:0 T:8 2358 XUs and 3462 XVs -> 332 XUVs
C:0 T:9 1023 XUs and 958 XVs -> 70 XUVs
C:0 T:10 343 XUs and 639 XVs -> 51 XUVs
C:0 T:11 995 XUs and 1286 XVs -> 81 XUVs
C:0 T:12 3781 XUs and 4733 XVs -> 329 XUVs
C:0 T:13 748 XUs and 1378 XVs -> 272 XUVs
C:0 T:14 4253 XUs and 6072 XVs -> 826 XUVs
C:0 T:15 4642 XUs and 5394 XVs -> 1205 XUVs
3667 XUVs total
1344 collection wire objects
3667 potential space points
Neighbour search...
254677 tests to find 127082 neighbours
Iterating with no regularization...
Begin: 1.60475e+09
0 1.56384e+09
1 1.56116e+09
2 1.56085e+09
Now with regularization...
Begin: 1.51954e+09
0 1.51941e+09
08-Sep-2025 16:40:57 BST Closed output file "np02vd_raw_run039324_2517_df-s05-d1_dw_0_20250907T210910_reco_stage1_20250908T154057_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 140.648 214.976 370.147 207.171 53.0299 26
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 7.556e-05 0.000106404 0.000219584 8.96455e-05 3.67778e-05 26
produce:tpcrawdecoder:PDVDTPCReader 21.2006 61.5614 136.753 58.0543 24.2126 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.0495151 0.0571809 0.0833725 0.0520855 0.00919992 26
produce:pdvddaphne:DAPHNEReaderPDVD 5.42596 8.48828 16.0962 8.09085 2.61705 26
produce:ophit:OpHitFinder 0.0362355 0.0437984 0.0530968 0.0435991 0.0037732 26
produce:opflash:OpFlashFinderVerticalDrift 0.00799901 0.0133732 0.0205354 0.0131098 0.00304865 26
produce:wclsdatavd:WireCellToolkit 51.646 64.7681 95.3334 61.3723 11.8593 26
produce:gaushit:GausHitFinder 0.705156 1.291 2.13389 1.29248 0.270539 26
produce:nhitsfilter:NumberOfHitsFilter 0.000313739 0.000467611 0.00059973 0.000468008 7.38572e-05 26
produce:reco3d:SpacePointSolver 6.59192 11.7943 20.1498 10.8674 3.40712 26
produce:hitpdune:DisambigFromSpacePoints 0.122452 0.235071 0.437477 0.229098 0.0780733 26
produce:pandora:StandardPandora 19.8292 61.2735 125.201 58.2273 25.5505 26
produce:pandoraTrack:LArPandoraTrackCreation 0.213756 1.44933 3.79286 1.10889 0.860673 26
produce:pandoraGnocalo:GnocchiCalorimetry 0.0148383 0.028861 0.0419378 0.029193 0.00779334 26
[art]:TriggerResults:TriggerResultInserter 2.6101e-05 4.48502e-05 0.000106323 4.4499e-05 1.55674e-05 26
end_path:out1:RootOutput 8.891e-06 1.27638e-05 3.8081e-05 1.14975e-05 5.27594e-06 26
end_path:out1:RootOutput(write) 3.32769 3.89989 5.74072 3.6358 0.720429 26
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4996.49 MB
Peak resident set size usage (VmHWM): 3100.97 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2517_df-s05-d1_dw_0_20250907T210910_reco_stage1_20250908T154057_keepup.root
\tHists: np02vd_raw_run039324_2517_df-s05-d1_dw_0_20250907T210910_reco_stage1_20250908T154058_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2517_df-s05-d1_dw_0_20250907T210910_reco_stage1_20250908T154057_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2517_df-s05-d1_dw_0_20250907T210910_reco_stage1_20250908T154057_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346059.0,
"core.end_time": 1757346059.0,
"core.events": [
1319295,
1319315,
1319335,
1319355,
1319375,
1319395,
1319415,
1319435,
1319455,
1319475,
1319495,
1319515,
1319535,
1319555,
1319575,
1319595,
1319615,
1319635,
1319655,
1319675,
1319695,
1319715,
1319735,
1319755,
1319775,
1319795
],
"core.event_count": 26,
"core.first_event_number": 1319295,
"core.last_event_number": 1319795,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2517_df-s05-d1_dw_0_20250907T210910.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2517_df-s05-d1_dw_0_20250907T210910_reco_stage1_20250908T154058_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346059.0,
"core.end_time": 1757346059.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2517_df-s05-d1_dw_0_20250907T210910.hdf5"
}
]
} 