How about use toLongExact instead of toIntExact? Seems if a parquet file is larger than 2GB, DiskRange.span will throw Interger Overflow. Io.: Error opening Hive split s3://xxxx/xxxx/000235_0 (offset=0, length=6678975885): integer overflowĪt io.ParquetPageSourceFactory.createParquetPageSource(ParquetPageSourceFactory.java:251)Īt io.ParquetPageSourceFactory.createPageSource(ParquetPageSourceFactory.java:129)Īt io.createHivePageSource(HivePageSourceProvider.java:188)Īt io.createPageSource(HivePageSourceProvider.java:108)Īt io.createPageSource(HivePageSourceProvider.java:91)Īt io.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:49)Īt io.(PageSourceManager.java:63)Īt io.$SplitToPages.process(ScanFilterAndProjectOperator.java:240)Īt io.$SplitToPages.process(ScanFilterAndProjectOperator.java:172)Īt io.$3.process(WorkProcessorUtils.java:319)Īt io.$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)Īt io.$3.process(WorkProcessorUtils.java:306)Īt io.(WorkProcessorUtils.java:221)Īt io.$processStateMonitor$2(WorkProcessorUtils.java:200)Īt io.$finishWhen$3(WorkProcessorUtils.java:215)Īt io.(WorkProcessorSourceOperatorAdapter.java:148)Īt io.(Driver.java:379)Īt io.$processFor$8(Driver.java:283)Īt io.(Driver.java:675)Īt io.(Driver.java:276)Īt io.$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)Īt io.process(PrioritizedSplitRunner.java:163)Īt io.$n(TaskExecutor.java:484)Īt io.prestosql.$gen.Presto_329_20200203_235109_2.run(Unknown Source)Īt .runWorker(ThreadPoolExecutor.java:1149)Īt $n(ThreadPoolExecutor.java:624)Ĭaused by: : integer overflowĪt (Math.java:1011)Īt io.(DiskRange.java:68)Īt io.rgeAdjacentDiskRanges(HdfsParquetDataSource.java:244)Īt io.HdfsParquetDataSource.readSmallDiskRanges(HdfsParquetDataSource.java:170)Īt io.anRead(HdfsParquetDataSource.java:158)Īt io.(ParquetReader.java:122)Īt io.ParquetPageSourceFactory.createParquetPageSource(ParquetPageSourceFactory.java:208)
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |