Jobsub ID 43071.133@dunegpschedd01.fnal.gov
| Jobsub ID | 43071.133@dunegpschedd01.fnal.gov |
| Workflow ID | 2329 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:39:55 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_brug |
| Last heartbeat | 2025-09-08 15:52:28 |
| From worker node | Hostname | wn-snel-020.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7H12 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:35:52 |
| Input files | vd-protodune:np02vd_raw_run039324_2805_df-s05-d0_dw_0_20250908T023751.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (4159s) |
| CPU time | 55m (3301s = 79%) |
| Max RSS bytes | 3131543552 (2986 MiB) |
| Outputting started | 2025-09-08 15:45:11 |
| Output files | |
| Finished | 2025-09-08 15:52:28 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
sp> EOS at call=53 with 8
[17:41:28.808] D [ glue ] frame sink sees EOS
[17:41:28.808] D [ pgraph ] <Pgrapher:> graph execution complete
[17:41:28.808] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.36 sec
[17:41:28.808] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.29 sec
[17:41:28.808] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.22 sec
[17:41:28.808] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.05 sec
[17:41:28.808] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.01 sec
[17:41:28.808] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.65 sec
[17:41:28.808] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.37 sec
[17:41:28.808] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.15 sec
[17:41:28.808] I [ timer ] Timer: WireCell::Aux::Resampler : 0.28 sec
[17:41:28.808] I [ timer ] Timer: WireCell::Aux::Resampler : 0.27 sec
[17:41:28.809] I [ timer ] Timer: WireCell::Aux::Resampler : 0.27 sec
[17:41:28.809] I [ timer ] Timer: WireCell::Aux::Resampler : 0.27 sec
[17:41:28.809] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[17:41:28.809] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:41:28.809] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:28.809] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:28.809] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:28.809] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:28.809] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:41:28.809] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:28.809] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:28.809] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:28.809] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[17:41:28.809] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:41:28.809] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:41:28.809] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:41:28.809] I [ timer ] Timer: Total node execution : 40.21000041440129 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 39253 traces tagged "gauss"
FrameSaver: q=6.85346e+06 n=865122 tag=gauss
wclsFrameSaver: saving 49085 traces tagged "wiener"
FrameSaver: q=7.4147e+06 n=833270 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 147 XUs and 289 XVs -> 3 XUVs
C:0 T:1 777 XUs and 469 XVs -> 35 XUVs
C:0 T:2 433 XUs and 697 XVs -> 28 XUVs
C:0 T:3 353 XUs and 471 XVs -> 8 XUVs
C:0 T:4 430 XUs and 456 XVs -> 13 XUVs
C:0 T:5 928 XUs and 830 XVs -> 28 XUVs
C:0 T:6 1011 XUs and 1114 XVs -> 62 XUVs
C:0 T:7 1396 XUs and 1410 XVs -> 66 XUVs
C:0 T:8 6333 XUs and 9620 XVs -> 2568 XUVs
C:0 T:9 9710 XUs and 13820 XVs -> 1888 XUVs
C:0 T:10 8942 XUs and 17021 XVs -> 2361 XUVs
C:0 T:11 137 XUs and 103 XVs -> 9 XUVs
C:0 T:12 9243 XUs and 17263 XVs -> 3122 XUVs
C:0 T:13 2755 XUs and 2324 XVs -> 148 XUVs
C:0 T:14 2421 XUs and 3649 XVs -> 133 XUVs
C:0 T:15 1646 XUs and 1336 XVs -> 91 XUVs
10563 XUVs total
1641 collection wire objects
10563 potential space points
Neighbour search...
1311559 tests to find 598892 neighbours
Iterating with no regularization...
Begin: 4.96408e+08
0 4.56783e+08
1 4.53768e+08
2 4.53089e+08
3 4.52848e+08
Now with regularization...
Begin: 4.42915e+08
0 4.42707e+08
08-Sep-2025 17:42:25 CEST Closed output file "np02vd_raw_run039324_2805_df-s05-d0_dw_0_20250908T023751_reco_stage1_20250908T154225_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 75.8833 146.421 226.401 145.053 36.802 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.833e-05 7.88411e-05 0.000180319 7.5301e-05 2.14052e-05 27
produce:tpcrawdecoder:PDVDTPCReader 12.9779 27.5412 43.9274 27.9305 7.74711 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.0342653 0.0346658 0.0353272 0.0346957 0.000238414 27
produce:pdvddaphne:DAPHNEReaderPDVD 3.57623 5.56803 8.38917 5.53809 1.28388 27
produce:ophit:OpHitFinder 0.0255366 0.0552323 0.10488 0.0530159 0.015489 27
produce:opflash:OpFlashFinderVerticalDrift 0.00549571 0.0150939 0.0258841 0.0150618 0.0044616 27
produce:wclsdatavd:WireCellToolkit 37.0086 43.8995 65.576 42.875 5.621 27
produce:gaushit:GausHitFinder 0.531307 1.08711 1.5971 1.07217 0.249366 27
produce:nhitsfilter:NumberOfHitsFilter 0.00015419 0.000331058 0.000769395 0.000303921 0.00012967 27
produce:reco3d:SpacePointSolver 4.52548 11.6748 19.8207 11.3545 3.49824 27
produce:hitpdune:DisambigFromSpacePoints 0.0733278 0.183163 0.321747 0.169375 0.0673597 27
produce:pandora:StandardPandora 9.63646 52.2426 114.881 46.5075 27.1467 27
produce:pandoraTrack:LArPandoraTrackCreation 0.2853 0.687919 1.90305 0.594947 0.336595 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.0139387 0.0244795 0.0424129 0.0240254 0.00619931 27
[art]:TriggerResults:TriggerResultInserter 1.4347e-05 2.2343e-05 4.5846e-05 1.7433e-05 8.24146e-06 27
end_path:out1:RootOutput 3.416e-06 4.54593e-06 1.6321e-05 3.887e-06 2.40119e-06 27
end_path:out1:RootOutput(write) 2.97026 3.35566 4.84795 3.24626 0.390739 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5010.09 MB
Peak resident set size usage (VmHWM): 3131.54 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2805_df-s05-d0_dw_0_20250908T023751_reco_stage1_20250908T154225_keepup.root
\tHists: np02vd_raw_run039324_2805_df-s05-d0_dw_0_20250908T023751_reco_stage1_20250908T154226_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2805_df-s05-d0_dw_0_20250908T023751_reco_stage1_20250908T154225_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2805_df-s05-d0_dw_0_20250908T023751_reco_stage1_20250908T154225_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346146.0,
"core.end_time": 1757346146.0,
"core.events": [
1471854,
1471874,
1471894,
1471914,
1471934,
1471954,
1471974,
1471994,
1472014,
1472034,
1472054,
1472074,
1472094,
1472114,
1472134,
1472154,
1472174,
1472194,
1472214,
1472234,
1472254,
1472274,
1472294,
1472314,
1472334,
1472354,
1472374
],
"core.event_count": 27,
"core.first_event_number": 1471854,
"core.last_event_number": 1472374,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2805_df-s05-d0_dw_0_20250908T023751.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2805_df-s05-d0_dw_0_20250908T023751_reco_stage1_20250908T154226_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346146.0,
"core.end_time": 1757346146.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2805_df-s05-d0_dw_0_20250908T023751.hdf5"
}
]
} 