Jobsub ID 36336.14@dunegpschedd02.fnal.gov
| Jobsub ID | 36336.14@dunegpschedd02.fnal.gov |
| Workflow ID | 2323 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:53:52 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-09-08 15:52:28 |
| From worker node | Hostname | n234.dur.scotgrid.ac.uk |
| cpuinfo | AMD EPYC 7352 24-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:07:04 |
| Input files | vd-protodune:np02vd_raw_run039324_1683_df-s05-d0_dw_0_20250907T034634.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5860s) |
| CPU time | 51m (3085s = 52%) |
| Max RSS bytes | 2906906624 (2772 MiB) |
| Outputting started | 2025-09-08 15:44:45 |
| Output files | |
| Finished | 2025-09-08 15:52:28 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
41:39.538] D [ glue ] frame sink sees EOS
[16:41:39.538] D [ pgraph ] <Pgrapher:> graph execution complete
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.59 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.14 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.5 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.34 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.26 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.11 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.95 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.93 sec
[16:41:39.538] I [ timer ] Timer: WireCell::Aux::Resampler : 0.27 sec
[16:41:39.538] I [ timer ] Timer: WireCell::Aux::Resampler : 0.27 sec
[16:41:39.538] I [ timer ] Timer: WireCell::Aux::Resampler : 0.27 sec
[16:41:39.538] I [ timer ] Timer: WireCell::Aux::Resampler : 0.26 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:41:39.538] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[16:41:39.538] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:39.538] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:39.538] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:39.538] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:39.538] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:41:39.538] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:41:39.538] I [ timer ] Timer: Total node execution : 36.92000067047775 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 29910 traces tagged "gauss"
FrameSaver: q=4.79657e+06 n=635017 tag=gauss
wclsFrameSaver: saving 37026 traces tagged "wiener"
FrameSaver: q=5.18338e+06 n=609567 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 515 XUs and 843 XVs -> 47 XUVs
C:0 T:1 1066 XUs and 1420 XVs -> 85 XUVs
C:0 T:2 1 XUs and 6 XVs -> 0 XUVs
C:0 T:3 576 XUs and 962 XVs -> 41 XUVs
C:0 T:4 742 XUs and 752 XVs -> 37 XUVs
C:0 T:5 154 XUs and 157 XVs -> 7 XUVs
C:0 T:6 68 XUs and 90 XVs -> 5 XUVs
C:0 T:7 344 XUs and 555 XVs -> 20 XUVs
C:0 T:8 12362 XUs and 21443 XVs -> 3997 XUVs
C:0 T:9 1430 XUs and 1582 XVs -> 65 XUVs
C:0 T:10 683 XUs and 837 XVs -> 38 XUVs
C:0 T:11 416 XUs and 419 XVs -> 17 XUVs
C:0 T:12 6670 XUs and 6928 XVs -> 784 XUVs
C:0 T:13 10176 XUs and 22830 XVs -> 2183 XUVs
C:0 T:14 140 XUs and 115 XVs -> 7 XUVs
C:0 T:15 156 XUs and 171 XVs -> 12 XUVs
7345 XUVs total
1104 collection wire objects
7345 potential space points
Neighbour search...
1035449 tests to find 437246 neighbours
Iterating with no regularization...
Begin: 3.09776e+08
0 2.65977e+08
1 2.63065e+08
2 2.62526e+08
3 2.62353e+08
Now with regularization...
Begin: 2.55345e+08
0 2.55181e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 16:42:23 BST Closed output file "np02vd_raw_run039324_1683_df-s05-d0_dw_0_20250907T034634_reco_stage1_20250908T154223_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 158.265 208.798 303.187 199.78 35.8577 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 4.6607e-05 6.25511e-05 9.1021e-05 5.9232e-05 1.20237e-05 27
produce:tpcrawdecoder:PDVDTPCReader 66.6926 90.4737 158.08 87.7003 21.8772 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.0589328 0.063607 0.0814024 0.0592199 0.00664176 27
produce:pdvddaphne:DAPHNEReaderPDVD 6.77817 9.48721 14.5605 8.70702 2.25138 27
produce:ophit:OpHitFinder 0.0278006 0.0356858 0.0419753 0.0359302 0.00349711 27
produce:opflash:OpFlashFinderVerticalDrift 0.0047368 0.00905345 0.0122489 0.00919925 0.00193546 27
produce:wclsdatavd:WireCellToolkit 33.5321 41.4832 61.2672 41.1781 5.66014 27
produce:gaushit:GausHitFinder 0.575352 0.917335 1.30797 0.885696 0.170915 27
produce:nhitsfilter:NumberOfHitsFilter 0.000142588 0.000251277 0.000424098 0.000233761 7.30956e-05 27
produce:reco3d:SpacePointSolver 6.01499 10.7124 15.9913 10.6955 2.96761 27
produce:hitpdune:DisambigFromSpacePoints 0.0742293 0.16319 0.254575 0.147374 0.0514101 27
produce:pandora:StandardPandora 17.115 51.4176 144.188 38.7112 28.8813 27
produce:pandoraTrack:LArPandoraTrackCreation 0.266619 0.553708 1.19375 0.539061 0.214417 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.0130958 0.0213984 0.0314082 0.021229 0.00442877 27
[art]:TriggerResults:TriggerResultInserter 1.0981e-05 1.53829e-05 4.765e-05 1.3927e-05 6.97646e-06 27
end_path:out1:RootOutput 2.866e-06 5.38841e-06 1.3596e-05 6.002e-06 2.31422e-06 27
end_path:out1:RootOutput(write) 3.12303 3.43181 4.94583 3.32472 0.420189 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4806.44 MB
Peak resident set size usage (VmHWM): 2906.91 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_1683_df-s05-d0_dw_0_20250907T034634_reco_stage1_20250908T154223_keepup.root
\tHists: np02vd_raw_run039324_1683_df-s05-d0_dw_0_20250907T034634_reco_stage1_20250908T154223_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_1683_df-s05-d0_dw_0_20250907T034634_reco_stage1_20250908T154223_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_1683_df-s05-d0_dw_0_20250907T034634_reco_stage1_20250908T154223_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346143.0,
"core.end_time": 1757346143.0,
"core.events": [
881734,
881754,
881774,
881794,
881814,
881834,
881854,
881874,
881894,
881914,
881934,
881954,
881974,
881994,
882014,
882034,
882054,
882074,
882094,
882114,
882134,
882154,
882174,
882194,
882214,
882234,
882254
],
"core.event_count": 27,
"core.first_event_number": 881734,
"core.last_event_number": 882254,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_1683_df-s05-d0_dw_0_20250907T034634.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_1683_df-s05-d0_dw_0_20250907T034634_reco_stage1_20250908T154223_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346143.0,
"core.end_time": 1757346143.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_1683_df-s05-d0_dw_0_20250907T034634.hdf5"
}
]
} 