Jobsub ID 237164.38@dunegpschedd02.fnal.gov
| Jobsub ID | 237164.38@dunegpschedd02.fnal.gov |
| Workflow ID | 9584 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-11-04 08:37:53 |
| Site | NL_SURFsara |
| Entry | DUNE_SurfSARA_arc02 |
| Last heartbeat | 2025-11-04 12:03:57 |
| From worker node | Hostname | wn-la-05.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-04 08:39:05 |
| Input files | vd-protodune:np02vd_raw_run040348_0789_df-s04-d3_dw_0_20251104T060429.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 3h (11701s) |
| CPU time | 3h (11020s = 94%) |
| Max RSS bytes | 5233467392 (4991 MiB) |
| Outputting started | 2025-11-04 11:54:08 |
| Output files | |
| Finished | 2025-11-04 12:03:57 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=45 anode=0
[12:53:04.452] D [ glue ] <FrameFanin:nfsp> EOS at call=45 with 8
[12:53:04.452] D [ glue ] frame sink sees EOS
[12:53:04.452] D [ pgraph ] <Pgrapher:> graph execution complete
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 15.12 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 13.55 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 13.48 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.84 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.21 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.12 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.35 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.26 sec
[12:53:04.453] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[12:53:04.453] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[12:53:04.453] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[12:53:04.453] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[12:53:04.453] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:53:04.453] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:53:04.453] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:53:04.453] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[12:53:04.453] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[12:53:04.453] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[12:53:04.453] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[12:53:04.453] I [ timer ] Timer: Total node execution : 102.9800003040582 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 40154 traces tagged "gauss"
FrameSaver: q=1.04684e+07 n=912900 tag=gauss
wclsFrameSaver: saving 50153 traces tagged "wiener"
FrameSaver: q=1.12413e+07 n=867687 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 98 XUs and 35 XVs -> 0 XUVs
C:0 T:1 208 XUs and 241 XVs -> 4 XUVs
C:0 T:2 229 XUs and 278 XVs -> 7 XUVs
C:0 T:3 697 XUs and 1110 XVs -> 25 XUVs
C:0 T:4 315 XUs and 418 XVs -> 14 XUVs
C:0 T:5 8785 XUs and 9136 XVs -> 654 XUVs
C:0 T:6 1084 XUs and 1376 XVs -> 79 XUVs
C:0 T:7 494 XUs and 673 XVs -> 18 XUVs
C:0 T:8 6759 XUs and 11373 XVs -> 1578 XUVs
C:0 T:9 9694 XUs and 18092 XVs -> 3244 XUVs
C:0 T:10 8946 XUs and 36728 XVs -> 4972 XUVs
C:0 T:11 1476 XUs and 5356 XVs -> 493 XUVs
C:0 T:12 9832 XUs and 17055 XVs -> 2923 XUVs
C:0 T:13 984 XUs and 1044 XVs -> 37 XUVs
C:0 T:14 313 XUs and 492 XVs -> 29 XUVs
C:0 T:15 503 XUs and 866 XVs -> 50 XUVs
14127 XUVs total
1475 collection wire objects
14127 potential space points
Neighbour search...
2242883 tests to find 928002 neighbours
Iterating with no regularization...
Begin: 4.21004e+09
0 3.98434e+09
1 3.97675e+09
2 3.97577e+09
Now with regularization...
Begin: 3.88933e+09
0 3.88868e+09
04-Nov-2025 12:53:53 CET Closed output file "np02vd_raw_run040348_0789_df-s04-d3_dw_0_20251104T060429_reco_stage1_20251104T115353_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 125.203 503.363 7691.36 174.926 1532.89 23
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.5517e-05 9.03613e-05 0.0002276 7.9388e-05 3.19165e-05 23
produce:tpcrawdecoder:PDVDTPCReader 12.9173 29.8534 65.7333 18.8293 17.0439 23
produce:triggerrawdecoder:PDVDTriggerReader4 0.545215 0.586418 0.711081 0.582253 0.0378923 23
produce:pdvddaphne:DAPHNEReaderPDVD 0.000274689 0.000361677 0.000675426 0.000348359 7.38396e-05 23
produce:ophit:OpHitFinder 6.9344e-05 0.000114892 0.000572081 8.7801e-05 0.000101786 23
produce:opflash:OpFlashFinderVerticalDrift 4.5607e-05 7.18183e-05 0.000308198 6.1142e-05 5.1572e-05 23
produce:wclsdatavd:WireCellToolkit 63.2821 102.538 239.634 98.2235 33.5181 23
produce:gaushit:GausHitFinder 0.814486 1.2808 3.86902 1.14581 0.598616 23
produce:nhitsfilter:NumberOfHitsFilter 0.00020164 0.000766152 0.00247553 0.000604321 0.000580574 23
produce:reco3d:SpacePointSolver 7.60006 16.8642 106.053 12.9598 19.2486 23
produce:hitpdune:DisambigFromSpacePoints 0.12895 0.27924 2.17951 0.188568 0.409116 23
produce:pandora:StandardPandora 12.3063 342.95 7258.67 24.1289 1474.58 23
produce:pandoraTrack:LArPandoraTrackCreation 0.768153 4.02858 54.4239 1.8208 10.7684 23
produce:pandoraGnocalo:GnocchiCalorimetry 0.0235459 0.0412604 0.122205 0.0381618 0.0184595 23
[art]:TriggerResults:TriggerResultInserter 2.5709e-05 5.35747e-05 0.000110314 5.0577e-05 1.58119e-05 23
end_path:out1:RootOutput 5.379e-06 1.08768e-05 5.4741e-05 8.563e-06 9.71304e-06 23
end_path:out1:RootOutput(write) 3.80725 4.80485 6.69152 4.93265 0.76778 23
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7132.05 MB
Peak resident set size usage (VmHWM): 5233.47 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040348_0789_df-s04-d3_dw_0_20251104T060429_reco_stage1_20251104T115353_keepup.root
\tHists: np02vd_raw_run040348_0789_df-s04-d3_dw_0_20251104T060429_reco_stage1_20251104T115354_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040348_0789_df-s04-d3_dw_0_20251104T060429_reco_stage1_20251104T115353_keepup.root
Ran successfully
{
"name": "np02vd_raw_run040348_0789_df-s04-d3_dw_0_20251104T060429_reco_stage1_20251104T115353_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_11_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1762257235.0,
"core.end_time": 1762257235.0,
"core.events": [
72887,
72891,
72895,
72899,
72903,
72907,
72911,
72915,
72919,
72923,
72927,
72931,
72935,
72939,
72943,
72947,
72951,
72955,
72959,
72963,
72967,
72971,
72975
],
"core.event_count": 23,
"core.first_event_number": 72887,
"core.last_event_number": 72975,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
40348
],
"core.runs_subruns": [
4034800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run040348_0789_df-s04-d3_dw_0_20251104T060429.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run040348_0789_df-s04-d3_dw_0_20251104T060429_reco_stage1_20251104T115354_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_11_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1762257235.0,
"core.end_time": 1762257235.0,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
40348
],
"core.runs_subruns": [
4034800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run040348_0789_df-s04-d3_dw_0_20251104T060429.hdf5"
}
]
} 