Jobsub ID 36377.134@dunegpschedd02.fnal.gov
| Jobsub ID | 36377.134@dunegpschedd02.fnal.gov |
| Workflow ID | 2327 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 13:33:57 |
| Site | ES_PIC |
| Entry | DUNE_T1_ES_PIC_ce15-multicore |
| Last heartbeat | 2025-09-08 15:52:34 |
| From worker node | Hostname | gpu07.pic.es |
| cpuinfo | AMD EPYC 9124 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:57:10 |
| Input files | vd-protodune:np02vd_raw_run039324_2345_df-s04-d2_dw_0_20250907T181059.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 45m (2738s) |
| CPU time | 35m (2109s = 77%) |
| Max RSS bytes | 3049242624 (2907 MiB) |
| Outputting started | 2025-09-08 15:42:49 |
| Output files | |
| Finished | 2025-09-08 15:52:34 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
.278] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.4 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.89 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.24 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 2.95 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 2.89 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 2.88 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 2.71 sec
[17:41:01.278] I [ timer ] Timer: WireCell::Aux::Resampler : 0.19 sec
[17:41:01.278] I [ timer ] Timer: WireCell::Aux::Resampler : 0.19 sec
[17:41:01.278] I [ timer ] Timer: WireCell::Aux::Resampler : 0.18 sec
[17:41:01.278] I [ timer ] Timer: WireCell::Aux::Resampler : 0.17 sec
[17:41:01.278] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[17:41:01.278] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:01.278] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:01.278] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:01.278] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:41:01.278] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:41:01.278] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:41:01.278] I [ timer ] Timer: Total node execution : 28.15000057592988 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 38523 traces tagged "gauss"
FrameSaver: q=7.03015e+06 n=900160 tag=gauss
wclsFrameSaver: saving 48369 traces tagged "wiener"
FrameSaver: q=7.60373e+06 n=866681 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 499 XUs and 673 XVs -> 18 XUVs
C:0 T:1 474 XUs and 661 XVs -> 29 XUVs
C:0 T:2 185 XUs and 290 XVs -> 9 XUVs
C:0 T:3 296 XUs and 364 XVs -> 15 XUVs
C:0 T:4 2857 XUs and 2433 XVs -> 196 XUVs
C:0 T:5 1065 XUs and 1280 XVs -> 90 XUVs
C:0 T:6 585 XUs and 614 XVs -> 16 XUVs
C:0 T:7 351 XUs and 493 XVs -> 34 XUVs
C:0 T:8 10284 XUs and 25530 XVs -> 2973 XUVs
C:0 T:9 7525 XUs and 10633 XVs -> 1519 XUVs
C:0 T:10 4438 XUs and 7346 XVs -> 2163 XUVs
C:0 T:11 2624 XUs and 3108 XVs -> 680 XUVs
C:0 T:12 4985 XUs and 4560 XVs -> 349 XUVs
C:0 T:13 6331 XUs and 11540 XVs -> 897 XUVs
C:0 T:14 15884 XUs and 33247 XVs -> 3517 XUVs
C:0 T:15 308 XUs and 268 XVs -> 19 XUVs
12524 XUVs total
1743 collection wire objects
12524 potential space points
Neighbour search...
1661002 tests to find 712396 neighbours
Iterating with no regularization...
Begin: 1.11059e+09
0 1.05723e+09
1 1.05369e+09
2 1.05283e+09
Now with regularization...
Begin: 1.04497e+09
0 1.04448e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 17:41:41 CEST Closed output file "np02vd_raw_run039324_2345_df-s04-d2_dw_0_20250907T181059_reco_stage1_20250908T154141_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 59.6006 97.408 211.457 87.9737 31.5566 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.2561e-05 8.55153e-05 0.000229684 7.5542e-05 3.52555e-05 27
produce:tpcrawdecoder:PDVDTPCReader 17.4461 20.4813 29.5572 19.6407 3.20104 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.0684156 0.0702939 0.0836759 0.0688056 0.00425276 27
produce:pdvddaphne:DAPHNEReaderPDVD 2.25504 2.78659 3.57732 2.73693 0.287744 27
produce:ophit:OpHitFinder 0.0254865 0.0310145 0.0367188 0.0303192 0.00271088 27
produce:opflash:OpFlashFinderVerticalDrift 0.0031567 0.00683767 0.00905653 0.006869 0.00125677 27
produce:wclsdatavd:WireCellToolkit 24.9281 29.3131 43.1575 28.4395 4.4286 27
produce:gaushit:GausHitFinder 0.374782 0.672058 0.974429 0.671464 0.16913 27
produce:nhitsfilter:NumberOfHitsFilter 0.000111762 0.000168048 0.000401878 0.000161133 5.22119e-05 27
produce:reco3d:SpacePointSolver 3.09788 8.08814 14.3202 7.76574 2.70573 27
produce:hitpdune:DisambigFromSpacePoints 0.0397828 0.120111 0.230226 0.112253 0.0491913 27
produce:pandora:StandardPandora 6.99127 32.7518 144.014 25.2655 26.0903 27
produce:pandoraTrack:LArPandoraTrackCreation 0.153796 0.49955 1.52986 0.39446 0.291357 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.00838516 0.01576 0.0265866 0.0152952 0.00427701 27
[art]:TriggerResults:TriggerResultInserter 1.368e-05 2.52436e-05 5.5141e-05 2.2301e-05 1.03163e-05 27
end_path:out1:RootOutput 2.81e-06 7.38489e-06 2.227e-05 5.81e-06 4.33857e-06 27
end_path:out1:RootOutput(write) 2.19985 2.5308 3.76652 2.43046 0.349652 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4967.84 MB
Peak resident set size usage (VmHWM): 3049.24 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2345_df-s04-d2_dw_0_20250907T181059_reco_stage1_20250908T154141_keepup.root
\tHists: np02vd_raw_run039324_2345_df-s04-d2_dw_0_20250907T181059_reco_stage1_20250908T154141_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2345_df-s04-d2_dw_0_20250907T181059_reco_stage1_20250908T154141_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2345_df-s04-d2_dw_0_20250907T181059_reco_stage1_20250908T154141_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346101.0,
"core.end_time": 1757346101.0,
"core.events": [
1228432,
1228452,
1228472,
1228492,
1228512,
1228532,
1228552,
1228572,
1228592,
1228612,
1228632,
1228652,
1228672,
1228692,
1228712,
1228732,
1228752,
1228772,
1228792,
1228812,
1228832,
1228852,
1228872,
1228892,
1228912,
1228932,
1228952
],
"core.event_count": 27,
"core.first_event_number": 1228432,
"core.last_event_number": 1228952,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2345_df-s04-d2_dw_0_20250907T181059.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2345_df-s04-d2_dw_0_20250907T181059_reco_stage1_20250908T154141_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346101.0,
"core.end_time": 1757346101.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2345_df-s04-d2_dw_0_20250907T181059.hdf5"
}
]
} 