Jobsub ID 253139.2@dunegpschedd02.fnal.gov
| Jobsub ID | 253139.2@dunegpschedd02.fnal.gov |
| Workflow ID | 10792 |
| Stage ID | 1 |
| User name | imawby@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 7200 (2 hours) |
| Submitted time | 2025-11-26 14:56:34 |
| Site | US_FNAL-FermiGrid |
| Entry | FNAL_GPGrid_ce03_mcore_op_duneonly |
| Last heartbeat | 2025-11-26 15:06:19 |
| From worker node | Hostname | dunegli-7735616-0-fnpc23035.fnal.gov |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 1048576000 (1000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-11-26 14:58:52 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1122_964_20230828T064340Z_gen_g4_detsim_hitreco__20240223T224918Z_reco2.root
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-11-26 15:06:19 |
| Saved logs | justin-logs:253139.2-dunegpschedd02.fnal.gov.logs.tgz |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
u_dune10kt_1x2x6_1122_964_20230828T064340Z_gen_g4_detsim_hitreco__20240223T224918Z_reco2.root
../justin-jobscript: line 64: 1305 Segmentation fault (core dumped) lar -c $FCL_FILE $events_option "$pfn" > ${fname}_reco_${now}.log 2>&1
lar exit code 139
=== Start last 100 lines of lar log file ===
StandardRawDigitExtractService::ctor: PedestalOption: 1
StandardRawDigitExtractService::ctor: FlagStuckOff: 0
StandardRawDigitExtractService::ctor: FlagStuckOn: 0
StandardRawDigitPrepService::ctor: Fetching extract service.
StandardRawDigitPrepService::ctor: Fetching channel status provider.
StandardRawDigitPrepService::ctor: Channel status provider: @0xc4cf8d0
StandardRawDigitPrepService::ctor: Extract service: @0xde38b80
StandardRawDigitPrepService::ctor: Fetching deconvolution service.
StandardRawDigitPrepService::ctor: Deconvolution service: @0x6fee170
StandardRawDigitPrepService::ctor: Fetching ROI building service.
StandardRawDigitPrepService::ctor: ROI building service: @0x25a8080
StandardRawDigitPrepService::ctor: Fetching wire building service.
StandardRawDigitPrepService::ctor: Wire building service: @0x25a8050
StandardRawDigitPrepService::ctor: StandardRawDigitPrepService:
StandardRawDigitPrepService::ctor: LogLevel: 1
StandardRawDigitPrepService::ctor: SkipBad: 1
StandardRawDigitPrepService::ctor: SkipNoisy: 0
StandardRawDigitPrepService::ctor: ChannelStatusOnline: 0
StandardRawDigitPrepService::ctor: DoMitigation: 0
StandardRawDigitPrepService::ctor: DoEarlySignalFinding: 0
StandardRawDigitPrepService::ctor: DoNoiseRemoval: 0
StandardRawDigitPrepService::ctor: DoDeconvolution: 1
StandardRawDigitPrepService::ctor: DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor: DoROI: 1
StandardRawDigitPrepService::ctor: DoWires: 1
StandardRawDigitPrepService::ctor: DoDump: 0
StandardRawDigitPrepService::ctor: DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
26-Nov-2025 14:59:31 UTC Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/43/4b/nu_dune10kt_1x2x6_1122_964_20230828T064340Z_gen_g4_detsim_hitreco__20240223T224918Z_reco2.root"
26-Nov-2025 14:59:33 UTC Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/43/4b/nu_dune10kt_1x2x6_1122_964_20230828T064340Z_gen_g4_detsim_hitreco__20240223T224918Z_reco2.root"
Begin processing the 1st record. run: 1122 subRun: 1 event: 96401 at 26-Nov-2025 14:59:33 UTC
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/WindowModel_U.pt'
Loaded the TorchScript model '/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/a0411fe586af02b8acadd6034a5179909d1beafe/SplitPosModel_U.pt'
PandoraMonitoring, only able to use default TApplication (limited functionality).
26-Nov-2025 14:59:39 UTC Opened output file with pattern "%ifb_reco2.root"
Begin processing the 2nd record. run: 1122 subRun: 1 event: 96402 at 26-Nov-2025 14:59:45 UTC
Begin processing the 3rd record. run: 1122 subRun: 1 event: 96403 at 26-Nov-2025 14:59:51 UTC
Begin processing the 4th record. run: 1122 subRun: 1 event: 96404 at 26-Nov-2025 14:59:53 UTC
Begin processing the 5th record. run: 1122 subRun: 1 event: 96405 at 26-Nov-2025 14:59:55 UTC
Begin processing the 6th record. run: 1122 subRun: 1 event: 96406 at 26-Nov-2025 14:59:57 UTC
Begin processing the 7th record. run: 1122 subRun: 1 event: 96407 at 26-Nov-2025 15:00:00 UTC
Begin processing the 8th record. run: 1122 subRun: 1 event: 96408 at 26-Nov-2025 15:00:02 UTC
Begin processing the 9th record. run: 1122 subRun: 1 event: 96409 at 26-Nov-2025 15:00:05 UTC
Begin processing the 10th record. run: 1122 subRun: 1 event: 96410 at 26-Nov-2025 15:00:07 UTC
Begin processing the 11th record. run: 1122 subRun: 1 event: 96411 at 26-Nov-2025 15:00:09 UTC
Begin processing the 12th record. run: 1122 subRun: 1 event: 96412 at 26-Nov-2025 15:00:10 UTC
Begin processing the 13th record. run: 1122 subRun: 1 event: 96413 at 26-Nov-2025 15:00:13 UTC
Begin processing the 14th record. run: 1122 subRun: 1 event: 96414 at 26-Nov-2025 15:00:16 UTC
Begin processing the 15th record. run: 1122 subRun: 1 event: 96415 at 26-Nov-2025 15:00:20 UTC
Begin processing the 16th record. run: 1122 subRun: 1 event: 96416 at 26-Nov-2025 15:00:22 UTC
Begin processing the 17th record. run: 1122 subRun: 1 event: 96417 at 26-Nov-2025 15:00:25 UTC
Begin processing the 18th record. run: 1122 subRun: 1 event: 96418 at 26-Nov-2025 15:00:27 UTC
Begin processing the 19th record. run: 1122 subRun: 1 event: 96419 at 26-Nov-2025 15:00:29 UTC
Begin processing the 20th record. run: 1122 subRun: 1 event: 96420 at 26-Nov-2025 15:00:33 UTC
Begin processing the 21st record. run: 1122 subRun: 1 event: 96421 at 26-Nov-2025 15:00:35 UTC
Begin processing the 22nd record. run: 1122 subRun: 1 event: 96422 at 26-Nov-2025 15:00:37 UTC
Begin processing the 23rd record. run: 1122 subRun: 1 event: 96423 at 26-Nov-2025 15:00:38 UTC
Begin processing the 24th record. run: 1122 subRun: 1 event: 96424 at 26-Nov-2025 15:00:40 UTC
Begin processing the 25th record. run: 1122 subRun: 1 event: 96425 at 26-Nov-2025 15:00:42 UTC
Begin processing the 26th record. run: 1122 subRun: 1 event: 96426 at 26-Nov-2025 15:00:46 UTC
Begin processing the 27th record. run: 1122 subRun: 1 event: 96427 at 26-Nov-2025 15:00:54 UTC
Begin processing the 28th record. run: 1122 subRun: 1 event: 96428 at 26-Nov-2025 15:00:57 UTC
Begin processing the 29th record. run: 1122 subRun: 1 event: 96429 at 26-Nov-2025 15:01:06 UTC
Begin processing the 30th record. run: 1122 subRun: 1 event: 96430 at 26-Nov-2025 15:01:08 UTC
Begin processing the 31st record. run: 1122 subRun: 1 event: 96431 at 26-Nov-2025 15:01:11 UTC
Begin processing the 32nd record. run: 1122 subRun: 1 event: 96432 at 26-Nov-2025 15:01:13 UTC
Begin processing the 33rd record. run: 1122 subRun: 1 event: 96433 at 26-Nov-2025 15:01:15 UTC
Begin processing the 34th record. run: 1122 subRun: 1 event: 96434 at 26-Nov-2025 15:01:17 UTC
Begin processing the 35th record. run: 1122 subRun: 1 event: 96435 at 26-Nov-2025 15:01:19 UTC
Begin processing the 36th record. run: 1122 subRun: 1 event: 96436 at 26-Nov-2025 15:01:21 UTC
Begin processing the 37th record. run: 1122 subRun: 1 event: 96437 at 26-Nov-2025 15:01:23 UTC
Begin processing the 38th record. run: 1122 subRun: 1 event: 96438 at 26-Nov-2025 15:01:25 UTC
Begin processing the 39th record. run: 1122 subRun: 1 event: 96439 at 26-Nov-2025 15:01:27 UTC
Begin processing the 40th record. run: 1122 subRun: 1 event: 96440 at 26-Nov-2025 15:01:29 UTC
Begin processing the 41st record. run: 1122 subRun: 1 event: 96441 at 26-Nov-2025 15:01:33 UTC
Begin processing the 42nd record. run: 1122 subRun: 1 event: 96442 at 26-Nov-2025 15:01:35 UTC
Begin processing the 43rd record. run: 1122 subRun: 1 event: 96443 at 26-Nov-2025 15:01:37 UTC
Begin processing the 44th record. run: 1122 subRun: 1 event: 96444 at 26-Nov-2025 15:01:39 UTC
Begin processing the 45th record. run: 1122 subRun: 1 event: 96445 at 26-Nov-2025 15:01:41 UTC
Begin processing the 46th record. run: 1122 subRun: 1 event: 96446 at 26-Nov-2025 15:01:44 UTC
Begin processing the 47th record. run: 1122 subRun: 1 event: 96447 at 26-Nov-2025 15:01:47 UTC
Begin processing the 48th record. run: 1122 subRun: 1 event: 96448 at 26-Nov-2025 15:01:50 UTC
Begin processing the 49th record. run: 1122 subRun: 1 event: 96449 at 26-Nov-2025 15:01:52 UTC
Begin processing the 50th record. run: 1122 subRun: 1 event: 96450 at 26-Nov-2025 15:01:54 UTC
Begin processing the 51st record. run: 1122 subRun: 1 event: 96451 at 26-Nov-2025 15:02:03 UTC
Begin processing the 52nd record. run: 1122 subRun: 1 event: 96452 at 26-Nov-2025 15:02:05 UTC
Begin processing the 53rd record. run: 1122 subRun: 1 event: 96453 at 26-Nov-2025 15:02:09 UTC
Begin processing the 54th record. run: 1122 subRun: 1 event: 96454 at 26-Nov-2025 15:02:12 UTC
=== End last 100 lines of lar log file ===
mv: cannot stat 'ClusterSplittingTraining.root': No such file or directory 