Jobsub ID 41052.155@dunegpschedd02.fnal.gov
| Jobsub ID | 41052.155@dunegpschedd02.fnal.gov |
| Workflow ID | 2815 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-17 14:45:21 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_dissel |
| Last heartbeat | 2025-09-17 16:18:15 |
| From worker node | Hostname | wn-lot-012.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | stalled |
| Started | 2025-09-17 15:28:11 |
| Input files | vd-protodune:np02vd_raw_run039388_0447_df-s05-d0_dw_0_20250916T223607.hdf5
|
| Outputting started | 2025-09-17 16:16:45 |
| Output files | |
| Finished | 2025-09-17 17:09:43 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
esampler:resmp0> EOS at call=37
[18:15:02.994] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=37 anode=0
[18:15:02.994] D [ glue ] <FrameFanin:nfsp> EOS at call=37 with 8
[18:15:02.994] D [ glue ] frame sink sees EOS
[18:15:02.994] D [ pgraph ] <Pgrapher:> graph execution complete
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.66 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.38 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.37 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.75 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.58 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.51 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.18 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.57 sec
[18:15:02.994] I [ timer ] Timer: WireCell::Aux::Resampler : 0.76 sec
[18:15:02.994] I [ timer ] Timer: WireCell::Aux::Resampler : 0.76 sec
[18:15:02.994] I [ timer ] Timer: WireCell::Aux::Resampler : 0.75 sec
[18:15:02.994] I [ timer ] Timer: WireCell::Aux::Resampler : 0.72 sec
[18:15:02.994] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:15:02.994] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:15:02.994] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:15:02.994] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:15:02.994] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:15:02.995] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[18:15:02.995] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[18:15:02.995] I [ timer ] Timer: Total node execution : 67.02000000886619 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 60760 traces tagged "gauss"
FrameSaver: q=1.03382e+07 n=1360206 tag=gauss
wclsFrameSaver: saving 73935 traces tagged "wiener"
FrameSaver: q=1.09962e+07 n=1313745 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 2653 XUs and 2769 XVs -> 57 XUVs
C:0 T:1 650 XUs and 880 XVs -> 27 XUVs
C:0 T:2 1832 XUs and 2319 XVs -> 66 XUVs
C:0 T:3 1066 XUs and 1216 XVs -> 47 XUVs
C:0 T:4 1330 XUs and 1403 XVs -> 105 XUVs
C:0 T:5 2054 XUs and 2029 XVs -> 206 XUVs
C:0 T:6 687 XUs and 771 XVs -> 27 XUVs
C:0 T:7 316 XUs and 507 XVs -> 20 XUVs
C:0 T:8 27333 XUs and 54590 XVs -> 10316 XUVs
C:0 T:9 8249 XUs and 11723 XVs -> 597 XUVs
C:0 T:10 1752 XUs and 3011 XVs -> 289 XUVs
C:0 T:11 1530 XUs and 2046 XVs -> 166 XUVs
C:0 T:12 1309 XUs and 2126 XVs -> 150 XUVs
C:0 T:13 1151 XUs and 1295 XVs -> 88 XUVs
C:0 T:14 1251 XUs and 1140 XVs -> 117 XUVs
C:0 T:15 1027 XUs and 967 XVs -> 65 XUVs
12343 XUVs total
2110 collection wire objects
12343 potential space points
Neighbour search...
1989209 tests to find 817008 neighbours
Iterating with no regularization...
Begin: 9.92831e+08
0 8.92346e+08
1 8.84963e+08
2 8.84241e+08
Now with regularization...
Begin: 8.71239e+08
0 8.70826e+08
17-Sep-2025 18:16:38 CEST Closed output file "np02vd_raw_run039388_0447_df-s05-d0_dw_0_20250916T223607_reco_stage1_20250917T161638_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 105.778 151.227 237.523 144.952 34.3936 19
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 3.8392e-05 5.57952e-05 8.9578e-05 5.8269e-05 1.19059e-05 19
produce:tpcrawdecoder:PDVDTPCReader 12.5246 14.5233 40.042 12.9959 6.03517 19
produce:triggerrawdecoder:PDVDTriggerReader4 0.288134 0.293788 0.319544 0.289509 0.00872801 19
produce:pdvddaphne:DAPHNEReaderPDVD 0.000196579 0.000286483 0.000599615 0.000230011 9.98652e-05 19
produce:ophit:OpHitFinder 8.1142e-05 0.000114085 0.000491251 8.7805e-05 9.04127e-05 19
produce:opflash:OpFlashFinderVerticalDrift 3.0417e-05 4.21296e-05 0.00021823 3.185e-05 4.1538e-05 19
produce:wclsdatavd:WireCellToolkit 52.8657 65.5608 81.4058 63.2399 7.63166 19
produce:gaushit:GausHitFinder 0.671508 1.06255 1.53148 1.01188 0.250379 19
produce:nhitsfilter:NumberOfHitsFilter 0.000172834 0.000317652 0.000654387 0.000289714 0.000115135 19
produce:reco3d:SpacePointSolver 6.71593 14.2105 23.6814 14.1644 4.7138 19
produce:hitpdune:DisambigFromSpacePoints 0.0929982 0.208061 0.44494 0.184784 0.0934302 19
produce:pandora:StandardPandora 19.2531 50.2956 110.355 47.2467 24.3845 19
produce:pandoraTrack:LArPandoraTrackCreation 0.357783 0.857572 1.75953 0.767743 0.386814 19
produce:pandoraGnocalo:GnocchiCalorimetry 0.0165139 0.0281616 0.051855 0.0256392 0.00847119 19
[art]:TriggerResults:TriggerResultInserter 1.1321e-05 1.38592e-05 3.7961e-05 1.2433e-05 5.74925e-06 19
end_path:out1:RootOutput 2.805e-06 3.87589e-06 1.4768e-05 3.316e-06 2.59146e-06 19
end_path:out1:RootOutput(write) 3.88537 4.15715 4.47829 4.14309 0.170814 19
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5026.19 MB
Peak resident set size usage (VmHWM): 3142.97 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039388_0447_df-s05-d0_dw_0_20250916T223607_reco_stage1_20250917T161638_keepup.root
\tHists: np02vd_raw_run039388_0447_df-s05-d0_dw_0_20250916T223607_reco_stage1_20250917T161638_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039388_0447_df-s05-d0_dw_0_20250916T223607_reco_stage1_20250917T161638_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039388_0447_df-s05-d0_dw_0_20250916T223607_reco_stage1_20250917T161638_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1758125798.0,
"core.end_time": 1758125798.0,
"core.events": [
34372,
34376,
34380,
34384,
34388,
34392,
34396,
34400,
34404,
34408,
34412,
34416,
34420,
34424,
34428,
34432,
34436,
34440,
34444
],
"core.event_count": 19,
"core.first_event_number": 34372,
"core.last_event_number": 34444,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39388
],
"core.runs_subruns": [
3938800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039388_0447_df-s05-d0_dw_0_20250916T223607.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039388_0447_df-s05-d0_dw_0_20250916T223607_reco_stage1_20250917T161638_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1758125798.0,
"core.end_time": 1758125798.0,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39388
],
"core.runs_subruns": [
3938800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039388_0447_df-s05-d0_dw_0_20250916T223607.hdf5"
}
]
} 