Jobsub ID 36317.190@dunegpschedd02.fnal.gov
| Jobsub ID | 36317.190@dunegpschedd02.fnal.gov |
| Workflow ID | 2326 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 10:47:50 |
| Site | UK_Glasgow |
| Entry | CLAS12_T3_UK_ScotGrid_GLA_ce04_scitok |
| Last heartbeat | 2025-09-08 15:52:36 |
| From worker node | Hostname | wn-d20-033.beowulf.cluster |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 13:35:06 |
| Input files | vd-protodune:np02vd_raw_run039324_2230_df-s01-d2_dw_0_20250907T145444.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (7115s) |
| CPU time | 1h (3734s = 52%) |
| Max RSS bytes | 2921066496 (2785 MiB) |
| Outputting started | 2025-09-08 15:33:42 |
| Output files | |
| Finished | 2025-09-08 15:52:36 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
[16:30:41.585] D [ glue ] <FrameFanin:nfsp> EOS at call=51 with 8
[16:30:41.585] D [ glue ] frame sink sees EOS
[16:30:41.585] D [ pgraph ] <Pgrapher:> graph execution complete
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.11 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.02 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.72 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.4 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.3 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.2 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.71 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.49 sec
[16:30:41.585] I [ timer ] Timer: WireCell::Aux::Resampler : 0.52 sec
[16:30:41.585] I [ timer ] Timer: WireCell::Aux::Resampler : 0.44 sec
[16:30:41.585] I [ timer ] Timer: WireCell::Aux::Resampler : 0.39 sec
[16:30:41.585] I [ timer ] Timer: WireCell::Aux::Resampler : 0.31 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[16:30:41.585] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:30:41.585] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:30:41.585] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:30:41.585] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[16:30:41.585] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:30:41.585] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:30:41.585] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:30:41.585] I [ timer ] Timer: Total node execution : 44.649999774992466 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 35943 traces tagged "gauss"
FrameSaver: q=6.03022e+06 n=704439 tag=gauss
wclsFrameSaver: saving 44308 traces tagged "wiener"
FrameSaver: q=6.36133e+06 n=668222 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 21 XUs and 20 XVs -> 1 XUVs
C:0 T:1 175 XUs and 217 XVs -> 18 XUVs
C:0 T:2 94 XUs and 127 XVs -> 3 XUVs
C:0 T:3 26 XUs and 39 XVs -> 1 XUVs
C:0 T:4 29 XUs and 25 XVs -> 0 XUVs
C:0 T:5 1270 XUs and 1357 XVs -> 34 XUVs
C:0 T:6 1053 XUs and 692 XVs -> 43 XUVs
C:0 T:7 1164 XUs and 1236 XVs -> 49 XUVs
C:0 T:8 6327 XUs and 10126 XVs -> 1888 XUVs
C:0 T:9 658 XUs and 1098 XVs -> 108 XUVs
C:0 T:10 225 XUs and 269 XVs -> 20 XUVs
C:0 T:11 2377 XUs and 3619 XVs -> 192 XUVs
C:0 T:12 4517 XUs and 5149 XVs -> 511 XUVs
C:0 T:13 5859 XUs and 12329 XVs -> 1118 XUVs
C:0 T:14 1494 XUs and 2863 XVs -> 209 XUVs
C:0 T:15 2995 XUs and 3294 XVs -> 133 XUVs
4328 XUVs total
1242 collection wire objects
4328 potential space points
Neighbour search...
501512 tests to find 225158 neighbours
Iterating with no regularization...
Begin: 6.38767e+08
0 6.00415e+08
1 5.94607e+08
2 5.94503e+08
Now with regularization...
Begin: 5.88051e+08
0 5.8798e+08
08-Sep-2025 16:31:39 BST Closed output file "np02vd_raw_run039324_2230_df-s01-d2_dw_0_20250907T145444_reco_stage1_20250908T153139_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 120.143 266.043 1782.65 195.943 307.097 26
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.5372e-05 0.000136752 0.000275569 0.000129825 4.42283e-05 26
produce:tpcrawdecoder:PDVDTPCReader 36.0751 119.698 1635.62 59.3757 303.485 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.057152 0.0639155 0.0902436 0.0580791 0.00954749 26
produce:pdvddaphne:DAPHNEReaderPDVD 4.37989 7.3853 12.0481 7.19558 2.22675 26
produce:ophit:OpHitFinder 0.0311492 0.0503777 0.0592215 0.0522736 0.00654253 26
produce:opflash:OpFlashFinderVerticalDrift 0.00645401 0.0169325 0.0264519 0.0170378 0.00467202 26
produce:wclsdatavd:WireCellToolkit 37.6762 50.8541 75.1122 49.0581 9.18706 26
produce:gaushit:GausHitFinder 0.685541 1.24455 2.11793 1.18942 0.348331 26
produce:nhitsfilter:NumberOfHitsFilter 0.000235044 0.00043628 0.00088487 0.00041905 0.000151601 26
produce:reco3d:SpacePointSolver 6.62383 13.677 21.006 12.623 4.38878 26
produce:hitpdune:DisambigFromSpacePoints 0.1114 0.230315 0.403795 0.193971 0.0979116 26
produce:pandora:StandardPandora 17.7517 67.0732 150.103 61.7197 33.8082 26
produce:pandoraTrack:LArPandoraTrackCreation 0.291227 1.07718 2.62562 0.946484 0.617885 26
produce:pandoraGnocalo:GnocchiCalorimetry 0.0158602 0.0344183 0.055889 0.0318247 0.0118282 26
[art]:TriggerResults:TriggerResultInserter 2.0498e-05 4.35306e-05 0.00012726 3.39335e-05 2.38976e-05 26
end_path:out1:RootOutput 4.408e-06 1.06258e-05 3.1069e-05 9.3675e-06 4.77609e-06 26
end_path:out1:RootOutput(write) 3.23992 4.56504 7.19997 4.60492 0.869515 26
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4856.85 MB
Peak resident set size usage (VmHWM): 2921.07 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2230_df-s01-d2_dw_0_20250907T145444_reco_stage1_20250908T153139_keepup.root
\tHists: np02vd_raw_run039324_2230_df-s01-d2_dw_0_20250907T145444_reco_stage1_20250908T153140_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2230_df-s01-d2_dw_0_20250907T145444_reco_stage1_20250908T153139_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2230_df-s01-d2_dw_0_20250907T145444_reco_stage1_20250908T153139_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757345500.0,
"core.end_time": 1757345500.0,
"core.events": [
1167700,
1167720,
1167740,
1167760,
1167780,
1167800,
1167820,
1167840,
1167860,
1167880,
1167900,
1167920,
1167940,
1167960,
1167980,
1168000,
1168020,
1168040,
1168060,
1168080,
1168100,
1168120,
1168140,
1168160,
1168180,
1168200
],
"core.event_count": 26,
"core.first_event_number": 1167700,
"core.last_event_number": 1168200,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2230_df-s01-d2_dw_0_20250907T145444.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2230_df-s01-d2_dw_0_20250907T145444_reco_stage1_20250908T153140_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757345500.0,
"core.end_time": 1757345500.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2230_df-s01-d2_dw_0_20250907T145444.hdf5"
}
]
} 