Jobsub ID 47593.9@dunegpschedd01.fnal.gov
| Jobsub ID | 47593.9@dunegpschedd01.fnal.gov |
| Workflow ID | 2815 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-17 14:47:21 |
| Site | CZ_FZU |
| Entry | Nova_CZ_FZU_osgce1 |
| Last heartbeat | 2025-09-17 17:12:44 |
| From worker node | Hostname | mahagon01.farm.particle.cz |
| cpuinfo | AMD EPYC 7301 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | stalled |
| Started | 2025-09-17 14:52:35 |
| Input files | vd-protodune:np02vd_raw_run039388_0342_df-s05-d0_dw_0_20250916T202308.hdf5
|
| Outputting started | 2025-09-17 16:25:09 |
| Output files | |
| Finished | 2025-09-17 18:10:49 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
] <OmnibusSigProc:anode0sigproc0> EOS at call=39 anode=0
[18:22:00.163] D [ glue ] <FrameFanin:nfsp> EOS at call=39 with 8
[18:22:00.163] D [ glue ] frame sink sees EOS
[18:22:00.163] D [ pgraph ] <Pgrapher:> graph execution complete
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 13.36 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.27 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.64 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.25 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.67 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.53 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.45 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.7 sec
[18:22:00.163] I [ timer ] Timer: WireCell::Aux::Resampler : 1.5 sec
[18:22:00.163] I [ timer ] Timer: WireCell::Aux::Resampler : 1.15 sec
[18:22:00.163] I [ timer ] Timer: WireCell::Aux::Resampler : 1.05 sec
[18:22:00.163] I [ timer ] Timer: WireCell::Aux::Resampler : 1.04 sec
[18:22:00.163] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:22:00.163] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:22:00.163] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:22:00.163] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[18:22:00.163] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:22:00.163] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[18:22:00.163] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[18:22:00.163] I [ timer ] Timer: Total node execution : 91.65999977476895 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 60201 traces tagged "gauss"
FrameSaver: q=9.96752e+06 n=1243943 tag=gauss
wclsFrameSaver: saving 72465 traces tagged "wiener"
FrameSaver: q=1.03737e+07 n=1199050 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 2806 XUs and 4131 XVs -> 122 XUVs
C:0 T:1 384 XUs and 459 XVs -> 10 XUVs
C:0 T:2 924 XUs and 614 XVs -> 42 XUVs
C:0 T:3 339 XUs and 262 XVs -> 12 XUVs
C:0 T:4 1031 XUs and 1276 XVs -> 34 XUVs
C:0 T:5 2133 XUs and 2176 XVs -> 58 XUVs
C:0 T:6 1811 XUs and 1876 XVs -> 66 XUVs
C:0 T:7 412 XUs and 444 XVs -> 13 XUVs
C:0 T:8 410 XUs and 727 XVs -> 35 XUVs
C:0 T:9 2379 XUs and 3698 XVs -> 121 XUVs
C:0 T:10 1065 XUs and 1815 XVs -> 59 XUVs
C:0 T:11 2234 XUs and 3002 XVs -> 136 XUVs
C:0 T:12 8674 XUs and 9759 XVs -> 289 XUVs
C:0 T:13 3625 XUs and 4595 XVs -> 216 XUVs
C:0 T:14 1309 XUs and 2350 XVs -> 107 XUVs
C:0 T:15 1693 XUs and 1662 XVs -> 91 XUVs
1411 XUVs total
1258 collection wire objects
1411 potential space points
Neighbour search...
7541 tests to find 3282 neighbours
Iterating with no regularization...
Begin: 3.59414e+08
0 3.51005e+08
1 3.50995e+08
Now with regularization...
Begin: 3.46282e+08
0 3.4628e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
17-Sep-2025 18:24:51 CEST Closed output file "np02vd_raw_run039388_0342_df-s05-d0_dw_0_20250916T202308_reco_stage1_20250917T162451_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 76.9476 273.569 496.392 256.143 86.6001 20
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 9.6052e-05 0.00013765 0.000332269 0.000116645 5.89572e-05 20
produce:tpcrawdecoder:PDVDTPCReader 9.19464 21.4341 30.9171 20.0336 5.03588 20
produce:triggerrawdecoder:PDVDTriggerReader4 0.27678 0.290864 0.36025 0.280857 0.0230226 20
produce:pdvddaphne:DAPHNEReaderPDVD 0.000446614 0.000575833 0.00158889 0.000520655 0.000239441 20
produce:ophit:OpHitFinder 0.000202844 0.000452832 0.00188213 0.000326332 0.000380322 20
produce:opflash:OpFlashFinderVerticalDrift 7.1004e-05 0.000143897 0.00078768 9.18085e-05 0.000153044 20
produce:wclsdatavd:WireCellToolkit 33.1824 113.781 167.295 112.276 26.0445 20
produce:gaushit:GausHitFinder 0.924718 2.1665 3.50828 2.1102 0.604281 20
produce:nhitsfilter:NumberOfHitsFilter 0.000185541 0.000492903 0.00109324 0.000435444 0.000174387 20
produce:reco3d:SpacePointSolver 3.9791 20.7584 40.7337 19.7451 7.75499 20
produce:hitpdune:DisambigFromSpacePoints 0.107083 0.376842 0.912643 0.319977 0.19973 20
produce:pandora:StandardPandora 26.4621 104.997 259.792 93.7618 56.6915 20
produce:pandoraTrack:LArPandoraTrackCreation 0.768358 2.42529 4.80439 2.3059 1.01804 20
produce:pandoraGnocalo:GnocchiCalorimetry 0.0216427 0.0553494 0.0889976 0.05306 0.0170967 20
[art]:TriggerResults:TriggerResultInserter 4.0897e-05 8.34819e-05 0.000253269 7.1986e-05 4.87766e-05 20
end_path:out1:RootOutput 6.322e-06 1.95032e-05 5.0827e-05 1.65905e-05 1.02423e-05 20
end_path:out1:RootOutput(write) 1.97013 7.21223 9.91845 6.97336 1.73499 20
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6350.52 MB
Peak resident set size usage (VmHWM): 4370.36 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039388_0342_df-s05-d0_dw_0_20250916T202308_reco_stage1_20250917T162451_keepup.root
\tHists: np02vd_raw_run039388_0342_df-s05-d0_dw_0_20250916T202308_reco_stage1_20250917T162451_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039388_0342_df-s05-d0_dw_0_20250916T202308_reco_stage1_20250917T162451_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039388_0342_df-s05-d0_dw_0_20250916T202308_reco_stage1_20250917T162451_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1758126293.0,
"core.end_time": 1758126293.0,
"core.events": [
26300,
26304,
26308,
26312,
26316,
26320,
26324,
26328,
26332,
26336,
26340,
26344,
26348,
26352,
26356,
26360,
26364,
26368,
26372,
26376
],
"core.event_count": 20,
"core.first_event_number": 26300,
"core.last_event_number": 26376,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39388
],
"core.runs_subruns": [
3938800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039388_0342_df-s05-d0_dw_0_20250916T202308.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039388_0342_df-s05-d0_dw_0_20250916T202308_reco_stage1_20250917T162451_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1758126293.0,
"core.end_time": 1758126293.0,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39388
],
"core.runs_subruns": [
3938800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039388_0342_df-s05-d0_dw_0_20250916T202308.hdf5"
}
]
} 