Jobsub ID 36335.49@dunegpschedd02.fnal.gov
| Jobsub ID | 36335.49@dunegpschedd02.fnal.gov |
| Workflow ID | 2327 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:47:55 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2025-09-08 15:52:30 |
| From worker node | Hostname | wn-sate-041.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7551P 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:05:09 |
| Input files | vd-protodune:np02vd_raw_run039324_2513_df-s05-d1_dw_0_20250907T210511.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5643s) |
| CPU time | 1h (4653s = 82%) |
| Max RSS bytes | 2821623808 (2690 MiB) |
| Outputting started | 2025-09-08 15:39:13 |
| Output files | |
| Finished | 2025-09-08 15:52:30 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
15] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.69 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.89 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.25 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.16 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.93 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.59 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.32 sec
[17:33:49.115] I [ timer ] Timer: WireCell::Aux::Resampler : 0.36 sec
[17:33:49.115] I [ timer ] Timer: WireCell::Aux::Resampler : 0.36 sec
[17:33:49.115] I [ timer ] Timer: WireCell::Aux::Resampler : 0.34 sec
[17:33:49.115] I [ timer ] Timer: WireCell::Aux::Resampler : 0.34 sec
[17:33:49.115] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:33:49.115] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:33:49.115] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:33:49.115] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:33:49.115] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:33:49.115] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:33:49.115] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:33:49.115] I [ timer ] Timer: Total node execution : 63.92000030167401 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 48044 traces tagged "gauss"
FrameSaver: q=1.52563e+07 n=1270696 tag=gauss
wclsFrameSaver: saving 60017 traces tagged "wiener"
FrameSaver: q=1.61075e+07 n=1228080 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 898 XUs and 1029 XVs -> 26 XUVs
C:0 T:1 18 XUs and 22 XVs -> 0 XUVs
C:0 T:2 14 XUs and 18 XVs -> 0 XUVs
C:0 T:3 674 XUs and 899 XVs -> 33 XUVs
C:0 T:4 97 XUs and 123 XVs -> 11 XUVs
C:0 T:5 263 XUs and 291 XVs -> 14 XUVs
C:0 T:6 1390 XUs and 1519 XVs -> 48 XUVs
C:0 T:7 160 XUs and 205 XVs -> 6 XUVs
C:0 T:8 25606 XUs and 55409 XVs -> 7713 XUVs
C:0 T:9 7469 XUs and 14279 XVs -> 2392 XUVs
C:0 T:10 12520 XUs and 33794 XVs -> 4977 XUVs
C:0 T:11 11469 XUs and 24902 XVs -> 2369 XUVs
C:0 T:12 26827 XUs and 60745 XVs -> 8037 XUVs
C:0 T:13 3084 XUs and 2344 XVs -> 92 XUVs
C:0 T:14 727 XUs and 857 XVs -> 49 XUVs
C:0 T:15 2006 XUs and 2726 XVs -> 114 XUVs
25881 XUVs total
2725 collection wire objects
25881 potential space points
Neighbour search...
4078665 tests to find 1623406 neighbours
Iterating with no regularization...
Begin: 6.21996e+09
0 5.80395e+09
1 5.75185e+09
2 5.75016e+09
Now with regularization...
Begin: 5.72326e+09
0 5.72243e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 17:38:46 CEST Closed output file "np02vd_raw_run039324_2513_df-s05-d1_dw_0_20250907T210511_reco_stage1_20250908T153846_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 123.431 206.12 407.417 171.35 73.5207 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.5613e-05 7.37228e-05 0.000165261 6.8398e-05 1.8627e-05 27
produce:tpcrawdecoder:PDVDTPCReader 17.226 34.7841 93.3695 32.2868 14.1915 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.0349425 0.0356882 0.0466081 0.0352673 0.00215053 27
produce:pdvddaphne:DAPHNEReaderPDVD 3.49723 5.04867 6.95177 4.88421 1.03259 27
produce:ophit:OpHitFinder 0.0365162 0.046876 0.0543726 0.0488264 0.00522913 27
produce:opflash:OpFlashFinderVerticalDrift 0.00766 0.01282 0.0186362 0.0136522 0.00279841 27
produce:wclsdatavd:WireCellToolkit 49.3652 57.8663 80.8319 56.4702 7.04993 27
produce:gaushit:GausHitFinder 0.762334 1.3924 1.9752 1.33195 0.313275 27
produce:nhitsfilter:NumberOfHitsFilter 0.000222669 0.000323623 0.000668669 0.000300445 8.21817e-05 27
produce:reco3d:SpacePointSolver 6.02428 14.3293 24.3068 13.7513 4.4591 27
produce:hitpdune:DisambigFromSpacePoints 0.106214 0.222782 0.434174 0.194408 0.0792054 27
produce:pandora:StandardPandora 32.9008 86.7594 282.486 62.972 63.8825 27
produce:pandoraTrack:LArPandoraTrackCreation 0.43505 1.06686 2.06181 0.883967 0.44906 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.0175092 0.0308455 0.0525958 0.0315759 0.00682625 27
[art]:TriggerResults:TriggerResultInserter 2.149e-05 3.88355e-05 0.000115557 3.4625e-05 1.91173e-05 27
end_path:out1:RootOutput 3.446e-06 1.0833e-05 4.263e-05 7.745e-06 7.45693e-06 27
end_path:out1:RootOutput(write) 3.93627 4.46263 6.18707 4.30511 0.502283 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4700.79 MB
Peak resident set size usage (VmHWM): 2821.62 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2513_df-s05-d1_dw_0_20250907T210511_reco_stage1_20250908T153846_keepup.root
\tHists: np02vd_raw_run039324_2513_df-s05-d1_dw_0_20250907T210511_reco_stage1_20250908T153847_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2513_df-s05-d1_dw_0_20250907T210511_reco_stage1_20250908T153846_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2513_df-s05-d1_dw_0_20250907T210511_reco_stage1_20250908T153846_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757345927.0,
"core.end_time": 1757345927.0,
"core.events": [
1317155,
1317175,
1317195,
1317215,
1317235,
1317255,
1317275,
1317295,
1317315,
1317335,
1317355,
1317375,
1317395,
1317415,
1317435,
1317455,
1317475,
1317495,
1317515,
1317535,
1317555,
1317575,
1317595,
1317615,
1317635,
1317655,
1317675
],
"core.event_count": 27,
"core.first_event_number": 1317155,
"core.last_event_number": 1317675,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2513_df-s05-d1_dw_0_20250907T210511.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2513_df-s05-d1_dw_0_20250907T210511_reco_stage1_20250908T153847_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757345927.0,
"core.end_time": 1757345927.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2513_df-s05-d1_dw_0_20250907T210511.hdf5"
}
]
} 