Jobsub ID 242968.94@dunegpschedd01.fnal.gov
| Jobsub ID | 242968.94@dunegpschedd01.fnal.gov |
| Workflow ID | 9584 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-11-04 13:50:09 |
| Site | NL_SURFsara |
| Entry | DUNE_SurfSARA_arc02 |
| Last heartbeat | 2025-11-04 16:05:48 |
| From worker node | Hostname | wn-da-16.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-04 13:51:04 |
| Input files | vd-protodune:np02vd_raw_run040348_1305_df-s04-d0_dw_0_20251104T131742.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7887s) |
| CPU time | 1h (5916s = 75%) |
| Max RSS bytes | 5868855296 (5596 MiB) |
| Outputting started | 2025-11-04 16:02:32 |
| Output files | |
| Finished | 2025-11-04 16:05:48 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
[sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=45 anode=0
[17:01:34.128] D [ glue ] <FrameFanin:nfsp> EOS at call=45 with 8
[17:01:34.128] D [ glue ] frame sink sees EOS
[17:01:34.128] D [ pgraph ] <Pgrapher:> graph execution complete
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 16.01 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 15.72 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 15.54 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 15.16 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 14.83 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 14.75 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 14.53 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 14.42 sec
[17:01:34.128] I [ timer ] Timer: WireCell::Aux::Resampler : 0.23 sec
[17:01:34.128] I [ timer ] Timer: WireCell::Aux::Resampler : 0.23 sec
[17:01:34.128] I [ timer ] Timer: WireCell::Aux::Resampler : 0.22 sec
[17:01:34.128] I [ timer ] Timer: WireCell::Aux::Resampler : 0.22 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:01:34.128] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:01:34.128] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:01:34.128] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:01:34.128] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:01:34.129] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:01:34.129] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:01:34.129] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:01:34.129] I [ timer ] Timer: Total node execution : 121.92000004276633 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 46706 traces tagged "gauss"
FrameSaver: q=1.18646e+07 n=979912 tag=gauss
wclsFrameSaver: saving 57464 traces tagged "wiener"
FrameSaver: q=1.26413e+07 n=933893 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 967 XUs and 893 XVs -> 26 XUVs
C:0 T:1 322 XUs and 414 XVs -> 14 XUVs
C:0 T:2 478 XUs and 342 XVs -> 16 XUVs
C:0 T:3 264 XUs and 289 XVs -> 13 XUVs
C:0 T:4 339 XUs and 824 XVs -> 14 XUVs
C:0 T:5 12697 XUs and 14081 XVs -> 950 XUVs
C:0 T:6 766 XUs and 912 XVs -> 40 XUVs
C:0 T:7 385 XUs and 358 XVs -> 21 XUVs
C:0 T:8 2426 XUs and 2260 XVs -> 119 XUVs
C:0 T:9 961 XUs and 989 XVs -> 53 XUVs
C:0 T:10 568 XUs and 358 XVs -> 39 XUVs
C:0 T:11 1166 XUs and 1098 XVs -> 43 XUVs
C:0 T:12 825 XUs and 548 XVs -> 33 XUVs
C:0 T:13 712 XUs and 1161 XVs -> 56 XUVs
C:0 T:14 1140 XUs and 1361 XVs -> 47 XUVs
C:0 T:15 3006 XUs and 3514 XVs -> 184 XUVs
1668 XUVs total
1173 collection wire objects
1668 potential space points
Neighbour search...
38388 tests to find 9902 neighbours
Iterating with no regularization...
Begin: 1.51622e+10
0 1.47688e+10
1 1.47569e+10
Now with regularization...
Begin: 1.46655e+10
0 1.46647e+10
04-Nov-2025 17:02:16 CET Closed output file "np02vd_raw_run040348_1305_df-s04-d0_dw_0_20251104T131742_reco_stage1_20251104T160216_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 138.488 339.883 1657.6 260.06 301.259 23
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.1439e-05 8.40886e-05 0.000230133 7.5516e-05 3.26144e-05 23
produce:tpcrawdecoder:PDVDTPCReader 30.9224 88.4294 208.654 75.4705 45.953 23
produce:triggerrawdecoder:PDVDTriggerReader4 0.598763 0.94838 1.69683 0.863215 0.271383 23
produce:pdvddaphne:DAPHNEReaderPDVD 0.000298012 0.000358896 0.000713731 0.000341657 8.08074e-05 23
produce:ophit:OpHitFinder 5.6839e-05 9.08023e-05 0.000500191 6.4383e-05 9.13358e-05 23
produce:opflash:OpFlashFinderVerticalDrift 4.1529e-05 6.22295e-05 0.000317831 4.5547e-05 5.61967e-05 23
produce:wclsdatavd:WireCellToolkit 56.1908 124.733 207.434 122.541 31.2491 23
produce:gaushit:GausHitFinder 0.775721 1.36566 2.43515 1.26322 0.405183 23
produce:nhitsfilter:NumberOfHitsFilter 0.000268496 0.000580907 0.00112858 0.000541481 0.000227229 23
produce:reco3d:SpacePointSolver 6.43575 14.1601 46.6857 10.9829 8.9087 23
produce:hitpdune:DisambigFromSpacePoints 0.0741985 0.256744 1.29221 0.172819 0.250793 23
produce:pandora:StandardPandora 10.7003 103.727 1322.68 24.8614 270.423 23
produce:pandoraTrack:LArPandoraTrackCreation 0.351038 1.65757 6.70509 1.15964 1.64337 23
produce:pandoraGnocalo:GnocchiCalorimetry 0.0221886 0.0365162 0.0582791 0.0337039 0.0101445 23
[art]:TriggerResults:TriggerResultInserter 2.1471e-05 3.76461e-05 7.4825e-05 2.8957e-05 1.52277e-05 23
end_path:out1:RootOutput 4.338e-06 6.87839e-06 2.611e-05 5.4e-06 4.47575e-06 23
end_path:out1:RootOutput(write) 4.01058 4.48349 5.16272 4.38264 0.379145 23
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7762.67 MB
Peak resident set size usage (VmHWM): 5868.86 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040348_1305_df-s04-d0_dw_0_20251104T131742_reco_stage1_20251104T160216_keepup.root
\tHists: np02vd_raw_run040348_1305_df-s04-d0_dw_0_20251104T131742_reco_stage1_20251104T160217_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040348_1305_df-s04-d0_dw_0_20251104T131742_reco_stage1_20251104T160216_keepup.root
Ran successfully
{
"name": "np02vd_raw_run040348_1305_df-s04-d0_dw_0_20251104T131742_reco_stage1_20251104T160216_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_11_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1762272138.0,
"core.end_time": 1762272138.0,
"core.events": [
120640,
120644,
120648,
120652,
120656,
120660,
120664,
120668,
120672,
120676,
120680,
120684,
120688,
120692,
120696,
120700,
120704,
120708,
120712,
120716,
120720,
120724,
120728
],
"core.event_count": 23,
"core.first_event_number": 120640,
"core.last_event_number": 120728,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
40348
],
"core.runs_subruns": [
4034800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run040348_1305_df-s04-d0_dw_0_20251104T131742.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run040348_1305_df-s04-d0_dw_0_20251104T131742_reco_stage1_20251104T160217_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_11_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1762272138.0,
"core.end_time": 1762272138.0,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
40348
],
"core.runs_subruns": [
4034800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run040348_1305_df-s04-d0_dw_0_20251104T131742.hdf5"
}
]
} 