Jobsub ID 36317.4@dunegpschedd02.fnal.gov
| Jobsub ID | 36317.4@dunegpschedd02.fnal.gov |
| Workflow ID | 2326 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 10:47:50 |
| Site | UK_Glasgow |
| Entry | CLAS12_T3_UK_ScotGrid_GLA_ce04_scitok |
| Last heartbeat | 2025-09-08 15:52:34 |
| From worker node | Hostname | wn-d20-007.beowulf.cluster |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 13:31:32 |
| Input files | vd-protodune:np02vd_raw_run039324_2222_df-s02-d2_dw_0_20250907T145021.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 2h (8283s) |
| CPU time | 1h (3613s = 43%) |
| Max RSS bytes | 3017580544 (2877 MiB) |
| Outputting started | 2025-09-08 15:49:36 |
| Output files | |
| Finished | 2025-09-08 15:52:34 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
4.162] D [ glue ] <FrameFanin:nfsp> EOS at call=51 with 8
[16:44:24.162] D [ glue ] frame sink sees EOS
[16:44:24.162] D [ pgraph ] <Pgrapher:> graph execution complete
[16:44:24.162] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.38 sec
[16:44:24.162] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.11 sec
[16:44:24.162] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.93 sec
[16:44:24.162] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.52 sec
[16:44:24.162] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.27 sec
[16:44:24.162] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.2 sec
[16:44:24.162] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.97 sec
[16:44:24.162] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.79 sec
[16:44:24.162] I [ timer ] Timer: WireCell::Aux::Resampler : 0.41 sec
[16:44:24.162] I [ timer ] Timer: WireCell::Aux::Resampler : 0.39 sec
[16:44:24.162] I [ timer ] Timer: WireCell::Aux::Resampler : 0.35 sec
[16:44:24.162] I [ timer ] Timer: WireCell::Aux::Resampler : 0.33 sec
[16:44:24.163] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[16:44:24.163] I [ timer ] Timer: WireCell::Gen::Retagger : 0.02 sec
[16:44:24.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:44:24.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:44:24.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:44:24.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:44:24.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:44:24.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:44:24.163] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:44:24.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:44:24.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:44:24.163] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:44:24.163] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:44:24.163] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:44:24.163] I [ timer ] Timer: Total node execution : 53.72999958693981 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 31195 traces tagged "gauss"
FrameSaver: q=5.0126e+06 n=670314 tag=gauss
wclsFrameSaver: saving 38543 traces tagged "wiener"
FrameSaver: q=5.31488e+06 n=645740 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 196 XUs and 247 XVs -> 11 XUVs
C:0 T:1 1584 XUs and 1629 XVs -> 56 XUVs
C:0 T:2 240 XUs and 324 XVs -> 10 XUVs
C:0 T:3 1391 XUs and 1323 XVs -> 65 XUVs
C:0 T:4 412 XUs and 504 XVs -> 30 XUVs
C:0 T:5 444 XUs and 367 XVs -> 10 XUVs
C:0 T:6 716 XUs and 685 XVs -> 20 XUVs
C:0 T:7 442 XUs and 501 XVs -> 13 XUVs
C:0 T:8 5456 XUs and 10347 XVs -> 2479 XUVs
C:0 T:9 2786 XUs and 4037 XVs -> 465 XUVs
C:0 T:10 2909 XUs and 5683 XVs -> 782 XUVs
C:0 T:11 385 XUs and 524 XVs -> 23 XUVs
C:0 T:12 4988 XUs and 6804 XVs -> 1071 XUVs
C:0 T:13 2561 XUs and 3378 XVs -> 728 XUVs
C:0 T:14 1156 XUs and 1337 XVs -> 67 XUVs
C:0 T:15 66 XUs and 79 XVs -> 10 XUVs
5840 XUVs total
1159 collection wire objects
5840 potential space points
Neighbour search...
825614 tests to find 358204 neighbours
Iterating with no regularization...
Begin: 5.29444e+08
0 4.96628e+08
1 4.94447e+08
2 4.94153e+08
Now with regularization...
Begin: 4.86146e+08
0 4.85991e+08
08-Sep-2025 16:45:15 BST Closed output file "np02vd_raw_run039324_2222_df-s02-d2_dw_0_20250907T145021_reco_stage1_20250908T154515_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 133.918 305.926 3313.08 177.829 602.568 26
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.482e-05 0.000148489 0.000401347 0.000131503 6.71185e-05 26
produce:tpcrawdecoder:PDVDTPCReader 17.0035 165.701 3174.95 47.8118 601.987 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.0568342 0.0600346 0.0708785 0.0574193 0.00495973 26
produce:pdvddaphne:DAPHNEReaderPDVD 3.69199 6.49061 11.9096 6.22912 2.15239 26
produce:ophit:OpHitFinder 0.0358244 0.0492416 0.0672833 0.0495252 0.00779116 26
produce:opflash:OpFlashFinderVerticalDrift 0.00951705 0.0150027 0.0258984 0.0141116 0.00408898 26
produce:wclsdatavd:WireCellToolkit 41.1799 57.3053 88.5096 54.1998 10.4595 26
produce:gaushit:GausHitFinder 0.635656 1.22246 1.86075 1.14651 0.33174 26
produce:nhitsfilter:NumberOfHitsFilter 0.000207461 0.00051778 0.00106632 0.000434514 0.000220888 26
produce:reco3d:SpacePointSolver 6.27489 12.6638 19.4415 12.9744 3.60678 26
produce:hitpdune:DisambigFromSpacePoints 0.0968885 0.214076 0.417962 0.206047 0.0812433 26
produce:pandora:StandardPandora 18.1811 56.9249 150.099 51.2257 30.5056 26
produce:pandoraTrack:LArPandoraTrackCreation 0.300747 0.863801 1.74809 0.85545 0.42598 26
produce:pandoraGnocalo:GnocchiCalorimetry 0.015134 0.0321858 0.0568985 0.032534 0.0112702 26
[art]:TriggerResults:TriggerResultInserter 2.0369e-05 5.31827e-05 0.000162998 3.9434e-05 3.95084e-05 26
end_path:out1:RootOutput 4.519e-06 1.19148e-05 3.3794e-05 9.8385e-06 6.55653e-06 26
end_path:out1:RootOutput(write) 3.33758 4.30971 6.73046 4.15722 0.844442 26
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4966.52 MB
Peak resident set size usage (VmHWM): 3017.58 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2222_df-s02-d2_dw_0_20250907T145021_reco_stage1_20250908T154515_keepup.root
\tHists: np02vd_raw_run039324_2222_df-s02-d2_dw_0_20250907T145021_reco_stage1_20250908T154515_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2222_df-s02-d2_dw_0_20250907T145021_reco_stage1_20250908T154515_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2222_df-s02-d2_dw_0_20250907T145021_reco_stage1_20250908T154515_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346316.0,
"core.end_time": 1757346316.0,
"core.events": [
1165364,
1165384,
1165404,
1165424,
1165444,
1165464,
1165484,
1165504,
1165524,
1165544,
1165564,
1165584,
1165604,
1165624,
1165644,
1165664,
1165684,
1165704,
1165724,
1165744,
1165764,
1165784,
1165804,
1165824,
1165844,
1165864
],
"core.event_count": 26,
"core.first_event_number": 1165364,
"core.last_event_number": 1165864,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2222_df-s02-d2_dw_0_20250907T145021.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2222_df-s02-d2_dw_0_20250907T145021_reco_stage1_20250908T154515_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346316.0,
"core.end_time": 1757346316.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2222_df-s02-d2_dw_0_20250907T145021.hdf5"
}
]
} 