Jobsub ID 257089.22@dunegpschedd02.fnal.gov
| Jobsub ID | 257089.22@dunegpschedd02.fnal.gov |
| Workflow ID | 10978 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| Requested | Processors | 2 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-12-03 21:35:46 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-12-04 00:47:06 |
| From worker node | Hostname | n164.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 2 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | stalled |
| Started | 2025-12-03 21:36:18 |
| Input files | vd-protodune:np02vd_raw_run041106_0542_df-s04-d0_dw_0_20251203T192734.hdf5
|
| Outputting started | 2025-12-04 00:44:38 |
| Output files | |
| Finished | 2025-12-04 01:26:56 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
igproc0> EOS at call=97 anode=0
[00:43:42.305] D [ glue ] <FrameFanin:nfsp> EOS at call=97 with 8
[00:43:42.305] D [ glue ] frame sink sees EOS
[00:43:42.305] D [ pgraph ] <Pgrapher:> graph execution complete
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 15.09 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 13.33 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.98 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.63 sec
[00:43:42.305] I [ timer ] Timer: WireCell::Aux::Resampler : 0.26 sec
[00:43:42.305] I [ timer ] Timer: WireCell::Aux::Resampler : 0.26 sec
[00:43:42.305] I [ timer ] Timer: WireCell::Aux::Resampler : 0.26 sec
[00:43:42.305] I [ timer ] Timer: WireCell::Aux::Resampler : 0.26 sec
[00:43:42.305] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[00:43:42.305] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:43:42.305] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[00:43:42.305] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[00:43:42.305] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[00:43:42.305] I [ timer ] Timer: Total node execution : 55.10999969393015 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 20506 traces tagged "gauss"
FrameSaver: q=6.59428e+06 n=494884 tag=gauss
wclsFrameSaver: saving 25978 traces tagged "wiener"
FrameSaver: q=7.34362e+06 n=478030 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 533 XUs and 722 XVs -> 32 XUVs
C:0 T:1 554 XUs and 413 XVs -> 6 XUVs
C:0 T:2 4826 XUs and 6364 XVs -> 410 XUVs
C:0 T:3 10260 XUs and 12037 XVs -> 856 XUVs
C:0 T:4 120 XUs and 119 XVs -> 7 XUVs
C:0 T:5 2557 XUs and 3031 XVs -> 144 XUVs
C:0 T:6 1257 XUs and 713 XVs -> 49 XUVs
C:0 T:7 514 XUs and 382 XVs -> 14 XUVs
1518 XUVs total
908 collection wire objects
1518 potential space points
Neighbour search...
62290 tests to find 14694 neighbours
Iterating with no regularization...
Begin: 5.87382e+09
0 5.67105e+09
1 5.66562e+09
Now with regularization...
Begin: 5.53265e+09
0 5.53225e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
04-Dec-2025 00:44:10 GMT Closed output file "np02vd_raw_run041106_0542_df-s04-d0_dw_0_20251203T192734_reco_stage1_20251204T004410_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 195.516 228.622 291.461 222.227 21.9705 49
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 4.9893e-05 6.42084e-05 0.000182369 5.9991e-05 1.85228e-05 49
produce:tpcrawdecoder:PDVDTPCReader 118.068 139.848 182.38 138.109 15.1258 49
produce:triggerrawdecoder:PDVDTriggerReader4 0.581669 0.913357 2.37391 0.82214 0.304147 49
produce:pdvddaphne:DAPHNEReaderPDVD 0.000388477 0.000432588 0.000826165 0.000410001 8.52192e-05 49
produce:ophit:OpHitFinder 5.1819e-05 7.69777e-05 0.00050731 6.3325e-05 6.71286e-05 49
produce:opflash:OpFlashFinderVerticalDrift 4.0129e-05 5.97772e-05 0.000312504 5.109e-05 4.14528e-05 49
produce:wclsdatavd:WireCellToolkit 47.1085 57.0808 86.6632 55.5043 8.01292 49
produce:gaushit:GausHitFinder 0.336888 0.774527 1.40066 0.71589 0.192666 49
produce:nhitsfilter:NumberOfHitsFilter 0.00012198 0.00021697 0.000470929 0.000199126 6.3594e-05 49
produce:reco3d:SpacePointSolver 2.08965 6.50147 12.773 5.89651 2.1905 49
produce:hitpdune:DisambigFromSpacePoints 0.0371716 0.0923325 0.250947 0.0860306 0.0350742 49
produce:pandora:StandardPandora 4.04738 20.7833 70.6112 17.9757 12.2515 49
produce:pandoraTrack:LArPandoraTrackCreation 0.126904 0.365353 1.43963 0.309225 0.212429 49
produce:pandoraGnocalo:GnocchiCalorimetry 0.00720754 0.0151637 0.0267402 0.0147788 0.00379513 49
[art]:TriggerResults:TriggerResultInserter 1.6535e-05 2.78284e-05 5.8017e-05 2.6829e-05 6.5858e-06 49
end_path:out1:RootOutput 2.607e-06 5.41306e-06 2.2081e-05 3.902e-06 3.09392e-06 49
end_path:out1:RootOutput(write) 1.96442 2.22058 2.93301 2.19115 0.16403 49
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4578.24 MB
Peak resident set size usage (VmHWM): 2541.53 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run041106_0542_df-s04-d0_dw_0_20251203T192734_reco_stage1_20251204T004410_keepup.root
\tHists: np02vd_raw_run041106_0542_df-s04-d0_dw_0_20251203T192734_reco_stage1_20251204T004410_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run041106_0542_df-s04-d0_dw_0_20251203T192734_reco_stage1_20251204T004410_keepup.root
Ran successfully
{
"name": "np02vd_raw_run041106_0542_df-s04-d0_dw_0_20251203T192734_reco_stage1_20251204T004410_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_13_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1764809051.0,
"core.end_time": 1764809051.0,
"core.events": [
26467,
26468,
26469,
26470,
26471,
26472,
26473,
26474,
26475,
26476,
26477,
26478,
26479,
26480,
26481,
26482,
26483,
26484,
26485,
26486,
26487,
26488,
26489,
26490,
26491,
26492,
26493,
26494,
26495,
26496,
26497,
26498,
26499,
26500,
26501,
26502,
26503,
26504,
26505,
26506,
26507,
26508,
26509,
26510,
26511,
26512,
26513,
26514,
26515
],
"core.event_count": 49,
"core.first_event_number": 26467,
"core.last_event_number": 26515,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
41106
],
"core.runs_subruns": [
4110600001
],
"dune.daq_test": true,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run041106_0542_df-s04-d0_dw_0_20251203T192734.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run041106_0542_df-s04-d0_dw_0_20251203T192734_reco_stage1_20251204T004410_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_13_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1764809051.0,
"core.end_time": 1764809051.0,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
41106
],
"core.runs_subruns": [
4110600001
],
"dune.daq_test": true,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run041106_0542_df-s04-d0_dw_0_20251203T192734.hdf5"
}
]
} 