Jobsub ID 36348.191@dunegpschedd02.fnal.gov
| Jobsub ID | 36348.191@dunegpschedd02.fnal.gov |
| Workflow ID | 2330 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 12:17:56 |
| Site | NL_SURFsara |
| Entry | DUNE_SurfSARA_arc01 |
| Last heartbeat | 2025-09-08 15:52:39 |
| From worker node | Hostname | wn-la-23.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:29:08 |
| Input files | vd-protodune:np02vd_raw_run039324_3014_df-s03-d0_dw_0_20250908T085604.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (4719s) |
| CPU time | 59m (3579s = 75%) |
| Max RSS bytes | 3112865792 (2968 MiB) |
| Outputting started | 2025-09-08 15:47:48 |
| Output files | |
| Finished | 2025-09-08 15:52:39 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
node0sigproc0> EOS at call=43 anode=0
[17:42:25.599] D [ glue ] <FrameFanin:nfsp> EOS at call=43 with 8
[17:42:25.599] D [ glue ] frame sink sees EOS
[17:42:25.599] D [ pgraph ] <Pgrapher:> graph execution complete
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.17 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.61 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.69 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.17 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.16 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.15 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.97 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.52 sec
[17:42:25.599] I [ timer ] Timer: WireCell::Aux::Resampler : 0.71 sec
[17:42:25.599] I [ timer ] Timer: WireCell::Aux::Resampler : 0.69 sec
[17:42:25.599] I [ timer ] Timer: WireCell::Aux::Resampler : 0.69 sec
[17:42:25.599] I [ timer ] Timer: WireCell::Aux::Resampler : 0.68 sec
[17:42:25.599] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:42:25.599] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:42:25.599] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:42:25.599] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[17:42:25.599] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:42:25.600] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:42:25.600] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:42:25.600] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:42:25.600] I [ timer ] Timer: Total node execution : 74.25999860651791 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 53290 traces tagged "gauss"
FrameSaver: q=8.02239e+06 n=1004230 tag=gauss
wclsFrameSaver: saving 66946 traces tagged "wiener"
FrameSaver: q=8.45898e+06 n=953644 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 628 XUs and 1059 XVs -> 56 XUVs
C:0 T:1 568 XUs and 588 XVs -> 81 XUVs
C:0 T:2 1213 XUs and 1153 XVs -> 70 XUVs
C:0 T:3 1195 XUs and 1057 XVs -> 92 XUVs
C:0 T:4 167 XUs and 249 XVs -> 16 XUVs
C:0 T:5 2248 XUs and 2195 XVs -> 85 XUVs
C:0 T:6 1549 XUs and 1602 XVs -> 110 XUVs
C:0 T:7 1846 XUs and 2155 XVs -> 102 XUVs
C:0 T:8 884 XUs and 1249 XVs -> 65 XUVs
C:0 T:9 1686 XUs and 1541 XVs -> 125 XUVs
C:0 T:10 1642 XUs and 2164 XVs -> 104 XUVs
C:0 T:11 2420 XUs and 2154 XVs -> 164 XUVs
C:0 T:12 2141 XUs and 2493 XVs -> 134 XUVs
C:0 T:13 2663 XUs and 2695 XVs -> 198 XUVs
C:0 T:14 4181 XUs and 3367 XVs -> 224 XUVs
C:0 T:15 779 XUs and 978 XVs -> 87 XUVs
1713 XUVs total
1375 collection wire objects
1713 potential space points
Neighbour search...
12305 tests to find 6700 neighbours
Iterating with no regularization...
Begin: 3.51016e+08
0 3.35028e+08
1 3.34257e+08
2 3.34237e+08
Now with regularization...
Begin: 3.27179e+08
0 3.27173e+08
08-Sep-2025 17:43:32 CEST Closed output file "np02vd_raw_run039324_3014_df-s03-d0_dw_0_20250908T085604_reco_stage1_20250908T154332_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 146.42 200.024 328.965 191.384 43.4191 22
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.9713e-05 8.64174e-05 0.000231236 7.89935e-05 3.22416e-05 22
produce:tpcrawdecoder:PDVDTPCReader 12.6876 41.2974 73.8712 42.8933 15.4371 22
produce:triggerrawdecoder:PDVDTriggerReader4 0.0371041 0.0418068 0.053 0.0403523 0.00480228 22
produce:pdvddaphne:DAPHNEReaderPDVD 4.00304 6.13552 9.26213 5.8464 1.2726 22
produce:ophit:OpHitFinder 0.0328737 0.0487065 0.0643867 0.0494572 0.00824137 22
produce:opflash:OpFlashFinderVerticalDrift 0.00591142 0.0108119 0.0188803 0.0112689 0.00310957 22
produce:wclsdatavd:WireCellToolkit 48.9277 71.8642 97.0562 72.4652 9.33505 22
produce:gaushit:GausHitFinder 0.852265 1.28326 1.81978 1.29819 0.257882 22
produce:nhitsfilter:NumberOfHitsFilter 0.000236153 0.000589787 0.00144498 0.000493948 0.000290376 22
produce:reco3d:SpacePointSolver 10.3098 15.9877 25.3871 16.0101 3.97884 22
produce:hitpdune:DisambigFromSpacePoints 0.124002 0.213292 0.394297 0.199358 0.0667345 22
produce:pandora:StandardPandora 24.7948 56.7652 184.688 48.4336 32.2349 22
produce:pandoraTrack:LArPandoraTrackCreation 0.349562 1.23612 3.09046 0.978007 0.672735 22
produce:pandoraGnocalo:GnocchiCalorimetry 0.0194805 0.029194 0.0436139 0.0270325 0.00703919 22
[art]:TriggerResults:TriggerResultInserter 2.3355e-05 3.65069e-05 7.344e-05 3.2088e-05 1.27772e-05 22
end_path:out1:RootOutput 5.929e-06 8.35514e-06 2.585e-05 7.0955e-06 4.11144e-06 22
end_path:out1:RootOutput(write) 3.61529 5.03426 6.81096 4.86801 0.725509 22
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4997.76 MB
Peak resident set size usage (VmHWM): 3112.87 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_3014_df-s03-d0_dw_0_20250908T085604_reco_stage1_20250908T154332_keepup.root
\tHists: np02vd_raw_run039324_3014_df-s03-d0_dw_0_20250908T085604_reco_stage1_20250908T154332_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_3014_df-s03-d0_dw_0_20250908T085604_reco_stage1_20250908T154332_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_3014_df-s03-d0_dw_0_20250908T085604_reco_stage1_20250908T154332_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346213.0,
"core.end_time": 1757346213.0,
"core.events": [
1579166,
1579186,
1579206,
1579226,
1579246,
1579266,
1579286,
1579306,
1579326,
1579346,
1579366,
1579386,
1579406,
1579426,
1579446,
1579466,
1579486,
1579506,
1579526,
1579546,
1579566,
1579586
],
"core.event_count": 22,
"core.first_event_number": 1579166,
"core.last_event_number": 1579586,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_3014_df-s03-d0_dw_0_20250908T085604.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_3014_df-s03-d0_dw_0_20250908T085604_reco_stage1_20250908T154332_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346213.0,
"core.end_time": 1757346213.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_3014_df-s03-d0_dw_0_20250908T085604.hdf5"
}
]
} 