Jobsub ID 39981.2@dunegpschedd02.fnal.gov
| Jobsub ID | 39981.2@dunegpschedd02.fnal.gov |
| Workflow ID | 2598 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-15 11:55:38 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-09-15 13:06:25 |
| From worker node | Hostname | wn-lot-014.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | stalled |
| Started | 2025-09-15 11:56:20 |
| Input files | vd-protodune:np02vd_raw_run039350_6997_df-s04-d1_dw_0_20250915T105042.hdf5
|
| Outputting started | 2025-09-15 13:05:11 |
| Output files | |
| Finished | 2025-09-15 13:17:54 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0
[15:03:58.348] D [ glue ] <FrameFanin:nfsp> EOS at call=35 with 8
[15:03:58.348] D [ glue ] frame sink sees EOS
[15:03:58.348] D [ pgraph ] <Pgrapher:> graph execution complete
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.52 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.33 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.24 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.52 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.48 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.36 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.23 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.18 sec
[15:03:58.349] I [ timer ] Timer: WireCell::Aux::Resampler : 0.81 sec
[15:03:58.349] I [ timer ] Timer: WireCell::Aux::Resampler : 0.8 sec
[15:03:58.349] I [ timer ] Timer: WireCell::Aux::Resampler : 0.79 sec
[15:03:58.349] I [ timer ] Timer: WireCell::Aux::Resampler : 0.78 sec
[15:03:58.349] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:03:58.349] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:03:58.349] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:03:58.349] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:03:58.349] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[15:03:58.349] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[15:03:58.349] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[15:03:58.349] I [ timer ] Timer: Total node execution : 69.07000013999641 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 43446 traces tagged "gauss"
FrameSaver: q=9.45624e+06 n=1012435 tag=gauss
wclsFrameSaver: saving 52624 traces tagged "wiener"
FrameSaver: q=9.85072e+06 n=977639 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 451 XUs and 595 XVs -> 28 XUVs
C:0 T:1 204 XUs and 292 XVs -> 5 XUVs
C:0 T:2 1024 XUs and 1330 XVs -> 77 XUVs
C:0 T:3 565 XUs and 475 XVs -> 17 XUVs
C:0 T:4 249 XUs and 206 XVs -> 9 XUVs
C:0 T:5 363 XUs and 269 XVs -> 11 XUVs
C:0 T:6 83 XUs and 158 XVs -> 2 XUVs
C:0 T:7 712 XUs and 1068 XVs -> 42 XUVs
C:0 T:8 3443 XUs and 5144 XVs -> 330 XUVs
C:0 T:9 2782 XUs and 2826 XVs -> 109 XUVs
C:0 T:10 10100 XUs and 25347 XVs -> 939 XUVs
C:0 T:11 2981 XUs and 3568 XVs -> 218 XUVs
C:0 T:12 1276 XUs and 2147 XVs -> 134 XUVs
C:0 T:13 1410 XUs and 1749 XVs -> 130 XUVs
C:0 T:14 371 XUs and 748 XVs -> 24 XUVs
C:0 T:15 616 XUs and 569 XVs -> 39 XUVs
2114 XUVs total
1537 collection wire objects
2114 potential space points
Neighbour search...
39674 tests to find 16426 neighbours
Iterating with no regularization...
Begin: 2.29976e+09
0 2.01139e+09
1 2.0087e+09
2 2.00855e+09
Now with regularization...
Begin: 1.99345e+09
0 1.99342e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
15-Sep-2025 15:05:03 CEST Closed output file "np02vd_raw_run039350_6997_df-s04-d1_dw_0_20250915T105042_reco_stage1_20250915T130503_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 169.671 226.996 283.447 223.695 30.908 18
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.3509e-05 8.18402e-05 0.000189525 7.43845e-05 2.7884e-05 18
produce:tpcrawdecoder:PDVDTPCReader 67.4583 94.7161 113.645 96.7931 13.3 18
produce:triggerrawdecoder:PDVDTriggerReader4 0.301759 0.389414 0.805888 0.347043 0.120396 18
produce:pdvddaphne:DAPHNEReaderPDVD 7.28558 8.96603 12.2698 8.7651 1.25106 18
produce:ophit:OpHitFinder 0.0527333 0.0625131 0.075961 0.0624623 0.00449808 18
produce:opflash:OpFlashFinderVerticalDrift 0.0131069 0.01801 0.026919 0.0170825 0.00380179 18
produce:wclsdatavd:WireCellToolkit 56.2419 66.7493 80.444 65.6634 6.99938 18
produce:gaushit:GausHitFinder 0.650876 1.08354 1.49973 1.11883 0.202291 18
produce:nhitsfilter:NumberOfHitsFilter 0.000232075 0.000608923 0.000893415 0.000655269 0.000208477 18
produce:reco3d:SpacePointSolver 7.26336 12.6137 19.7146 12.1823 3.23652 18
produce:hitpdune:DisambigFromSpacePoints 0.07238 0.162665 0.250979 0.160262 0.04725 18
produce:pandora:StandardPandora 14.6093 36.7449 56.8628 39.8596 12.1616 18
produce:pandoraTrack:LArPandoraTrackCreation 0.357121 0.774769 1.38584 0.774074 0.259017 18
produce:pandoraGnocalo:GnocchiCalorimetry 0.0176753 0.0275163 0.0378279 0.0272313 0.00558548 18
[art]:TriggerResults:TriggerResultInserter 1.5539e-05 2.01656e-05 4.9873e-05 1.88805e-05 7.41614e-06 18
end_path:out1:RootOutput 3.908e-06 5.83989e-06 1.9817e-05 5.1145e-06 3.44155e-06 18
end_path:out1:RootOutput(write) 4.33708 4.64174 4.89702 4.62983 0.160363 18
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4804.64 MB
Peak resident set size usage (VmHWM): 2919.85 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039350_6997_df-s04-d1_dw_0_20250915T105042_reco_stage1_20250915T130503_keepup.root
\tHists: np02vd_raw_run039350_6997_df-s04-d1_dw_0_20250915T105042_reco_stage1_20250915T130503_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039350_6997_df-s04-d1_dw_0_20250915T105042_reco_stage1_20250915T130503_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039350_6997_df-s04-d1_dw_0_20250915T105042_reco_stage1_20250915T130503_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757941504.0,
"core.end_time": 1757941504.0,
"core.events": [
510609,
510613,
510617,
510621,
510625,
510629,
510633,
510637,
510641,
510645,
510649,
510653,
510657,
510661,
510665,
510669,
510673,
510677
],
"core.event_count": 18,
"core.first_event_number": 510609,
"core.last_event_number": 510677,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39350
],
"core.runs_subruns": [
3935000001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039350_6997_df-s04-d1_dw_0_20250915T105042.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039350_6997_df-s04-d1_dw_0_20250915T105042_reco_stage1_20250915T130503_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757941504.0,
"core.end_time": 1757941504.0,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39350
],
"core.runs_subruns": [
3935000001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039350_6997_df-s04-d1_dw_0_20250915T105042.hdf5"
}
]
} 