Jobsub ID 45371.90@dunegpschedd01.fnal.gov
| Jobsub ID | 45371.90@dunegpschedd01.fnal.gov |
| Workflow ID | 2521 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-11 22:00:09 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2025-09-12 00:35:15 |
| From worker node | Hostname | wn-sate-044.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7551P 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-11 22:30:15 |
| Input files | vd-protodune:np02vd_raw_run039273_0587_df-s01-d2_dw_0_20250831T170433.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (7146s) |
| CPU time | 1h (5593s = 78%) |
| Max RSS bytes | 3330666496 (3176 MiB) |
| Outputting started | 2025-09-12 00:29:22 |
| Output files | |
| Finished | 2025-09-12 00:35:15 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
sp> EOS at call=35 with 8
[02:25:45.749] D [ glue ] frame sink sees EOS
[02:25:45.749] D [ pgraph ] <Pgrapher:> graph execution complete
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 15.27 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.6 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.84 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.78 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.23 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.14 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.58 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.49 sec
[02:25:45.749] I [ timer ] Timer: WireCell::Aux::Resampler : 1.26 sec
[02:25:45.749] I [ timer ] Timer: WireCell::Aux::Resampler : 1.24 sec
[02:25:45.749] I [ timer ] Timer: WireCell::Aux::Resampler : 1.22 sec
[02:25:45.749] I [ timer ] Timer: WireCell::Aux::Resampler : 1.22 sec
[02:25:45.749] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[02:25:45.749] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[02:25:45.749] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[02:25:45.749] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[02:25:45.749] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[02:25:45.749] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[02:25:45.749] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[02:25:45.749] I [ timer ] Timer: Total node execution : 94.97000036202371 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 58214 traces tagged "gauss"
FrameSaver: q=1.11242e+07 n=1403154 tag=gauss
wclsFrameSaver: saving 72989 traces tagged "wiener"
FrameSaver: q=1.18029e+07 n=1351385 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 462 XUs and 645 XVs -> 36 XUVs
C:0 T:1 595 XUs and 1325 XVs -> 37 XUVs
C:0 T:2 594 XUs and 766 XVs -> 28 XUVs
C:0 T:3 891 XUs and 900 XVs -> 40 XUVs
C:0 T:4 256 XUs and 333 XVs -> 8 XUVs
C:0 T:5 1064 XUs and 843 XVs -> 34 XUVs
C:0 T:6 1389 XUs and 2487 XVs -> 126 XUVs
C:0 T:7 865 XUs and 1112 XVs -> 56 XUVs
C:0 T:8 24358 XUs and 46213 XVs -> 10025 XUVs
C:0 T:9 11460 XUs and 18479 XVs -> 2494 XUVs
C:0 T:10 8022 XUs and 22345 XVs -> 3004 XUVs
C:0 T:11 2934 XUs and 3760 XVs -> 296 XUVs
C:0 T:12 6090 XUs and 8615 XVs -> 582 XUVs
C:0 T:13 2363 XUs and 4386 XVs -> 392 XUVs
C:0 T:14 4511 XUs and 8041 XVs -> 904 XUVs
C:0 T:15 2969 XUs and 2860 XVs -> 317 XUVs
18379 XUVs total
2652 collection wire objects
18379 potential space points
Neighbour search...
2879487 tests to find 1191666 neighbours
Iterating with no regularization...
Begin: 7.45751e+08
0 6.65759e+08
1 6.61039e+08
2 6.60324e+08
3 6.60079e+08
Now with regularization...
Begin: 6.45793e+08
0 6.45539e+08
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
12-Sep-2025 02:29:10 CEST Closed output file "np02vd_raw_run039273_0587_df-s01-d2_dw_0_20250831T170433_reco_stage1_20250912T002910_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 236.923 393.265 563.715 384.754 85.9995 18
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.1466e-05 8.64241e-05 0.000295745 7.37685e-05 5.22515e-05 18
produce:tpcrawdecoder:PDVDTPCReader 55.1422 80.3345 108.395 75.3932 15.2935 18
produce:triggerrawdecoder:PDVDTriggerReader4 0.0347757 0.0420312 0.0800212 0.0397237 0.0104227 18
produce:pdvddaphne:DAPHNEReaderPDVD 9.13749 12.8281 16.3692 12.7239 2.04497 18
produce:ophit:OpHitFinder 0.0608828 0.0684514 0.0788605 0.0668615 0.00478826 18
produce:opflash:OpFlashFinderVerticalDrift 0.0138003 0.0196547 0.025938 0.0189359 0.00282082 18
produce:wclsdatavd:WireCellToolkit 90.9829 100.453 109.535 100.264 5.44763 18
produce:gaushit:GausHitFinder 1.453 2.37295 3.10471 2.42329 0.462629 18
produce:nhitsfilter:NumberOfHitsFilter 0.000379913 0.000517737 0.000816313 0.000482812 0.000106744 18
produce:reco3d:SpacePointSolver 12.2877 25.3212 38.0254 26.1204 6.74488 18
produce:hitpdune:DisambigFromSpacePoints 0.21672 0.463508 0.71899 0.447866 0.141977 18
produce:pandora:StandardPandora 45.7018 160.632 329.148 152.7 78.7674 18
produce:pandoraTrack:LArPandoraTrackCreation 1.20981 3.11302 5.74013 3.05023 1.21391 18
produce:pandoraGnocalo:GnocchiCalorimetry 0.0444566 0.0572844 0.0786518 0.0568669 0.00853169 18
[art]:TriggerResults:TriggerResultInserter 2.2723e-05 3.95065e-05 0.000142397 3.1189e-05 2.69938e-05 18
end_path:out1:RootOutput 4.719e-06 1.03886e-05 4.5686e-05 8.872e-06 8.70931e-06 18
end_path:out1:RootOutput(write) 6.93292 7.49683 8.20593 7.50589 0.301955 18
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5211.37 MB
Peak resident set size usage (VmHWM): 3330.67 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039273_0587_df-s01-d2_dw_0_20250831T170433_reco_stage1_20250912T002910_keepup.root
\tHists: np02vd_raw_run039273_0587_df-s01-d2_dw_0_20250831T170433_reco_stage1_20250912T002911_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039273_0587_df-s01-d2_dw_0_20250831T170433_reco_stage1_20250912T002910_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039273_0587_df-s01-d2_dw_0_20250831T170433_reco_stage1_20250912T002910_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757636951.0,
"core.end_time": 1757636951.0,
"core.events": [
211340,
211360,
211380,
211400,
211420,
211440,
211460,
211480,
211500,
211520,
211540,
211560,
211580,
211600,
211620,
211640,
211660,
211680
],
"core.event_count": 18,
"core.first_event_number": 211340,
"core.last_event_number": 211680,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39273
],
"core.runs_subruns": [
3927300001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039273_0587_df-s01-d2_dw_0_20250831T170433.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039273_0587_df-s01-d2_dw_0_20250831T170433_reco_stage1_20250912T002911_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757636951.0,
"core.end_time": 1757636951.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39273
],
"core.runs_subruns": [
3927300001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039273_0587_df-s01-d2_dw_0_20250831T170433.hdf5"
}
]
} 