• Title/Summary/Keyword: Data loading

Search Result 2,098, Processing Time 0.048 seconds

Analysis of the Influence Factors of Data Loading Performance Using Apache Sqoop (아파치 스쿱을 사용한 하둡의 데이터 적재 성능 영향 요인 분석)

  • Chen, Liu;Ko, Junghyun;Yeo, Jeongmo
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.4 no.2
    • /
    • pp.77-82
    • /
    • 2015
  • Big Data technology has been attracted much attention in aspect of fast data processing. Research of practicing Big Data technology is also ongoing to process large-scale structured data much faster in Relatioinal Database(RDB). Although there are lots of studies about measuring analyzing performance, studies about structured data loading performance, prior step of analyzing, is very rare. Thus, in this study, structured data in RDB is tested the performance that loads distributed processing platform Hadoop using Apache sqoop. Also in order to analyze the influence factors of data loading, it is tested repeatedly with different options of data loading and compared with data loading performance among RDB based servers. Although data loading performance of Apache Sqoop in test environment was low, but in large-scale Hadoop cluster environment we can expect much better performance because of getting more hardware resources. It is expected to be based on study improving data loading performance and whole steps of performance analyzing structured data in Hadoop Platform.

Prediction of Settlement Based on Field Monitoring Data under Preloading Improvement with Ramp Loading

  • Woo, Sang-Inn;Yune, Chan-Young;Chung, Choong-Ki
    • Proceedings of the Korean Geotechical Society Conference
    • /
    • 2008.10a
    • /
    • pp.436-452
    • /
    • 2008
  • In this study, the settlement prediction method based on field monitoring data under preloading improvement with ramp loading is developed. Settlement behavior can be predicted with field monitored settlement throughout the entire preloading process including ramp loading followed by constant loading. The developed method is verified by comparing its predicted results with results from physical model tests and field monitoring data.

  • PDF

Compensation of Aethalometer Black Carbon Data Observed at a Gwangju Site (광주 도심지역에서 측정한 Aethalometer 검댕입자 자료의 보정)

  • Park, Seung-Shik;Jung, Jung-H.;Cho, Sung-Y.;Kim, Seung-Jai
    • Journal of Korean Society for Atmospheric Environment
    • /
    • v.25 no.6
    • /
    • pp.571-578
    • /
    • 2009
  • $PM_{2.5}$ black carbon (BC) concentrations were measured to investigate the filter spot loading effect in raw BC data at 5-minute time-based resolution using a single-wavelength aethalometer at a Gwangju site. Also the elemental carbon (EC) concentrations from 24-hr integrated filter-based measurements of $PM_{2.5}$ particles were determined to compare with the loading compensated BC values. Close examination of the time-series BC data showed clearly the "gaps" when the filter tape advances, suggesting the correction of raw BC data. Therefore, we calculated the average BC concentration in each range of attenuation (ATN) to decide if there was (or was not) an effect on the aethalometer data according to the loading of the filter spot. A consistent decrease of average BC concentration was found with increasing ATN values for every month, suggesting there was a consistent "spot loading effect" in the raw BC data. The loading compensated BC concentration according to a simple compensation model with loading effect was 1.01~1.15 times greater than the raw BC data. The 24-hr average concentration of EC observed during summer sampling period was about 3% higher than the original 24-hr average BC value and 2% lower than the loading compensated BC concentration.

Analysis of Factors Affecting Peak Loading Coefficient of Sewer Works in Korea (우리나라 하수도시설의 첨두부하율 영향요소 분석)

  • Hyun, In-Hwan;Lee, Young-Ho
    • Journal of Korean Society of Water and Wastewater
    • /
    • v.25 no.6
    • /
    • pp.877-884
    • /
    • 2011
  • Although peak loading coefficient is one of critical design factors for sewer works, its detailed affecting factors were not analyzed because of limited data availability. This study analyzed the affecting factors on peak loading coefficient with plenty data obtained from several newly constructed sewer works. Simple and multiple regression analysis methods were adopted to analyze the relationships of each variable with or without data filtering. Drainage population, drainage area, population density, and daily sewage flow per person showed very weak relationships under diverse characteristics of cities. However, daily sewage flow per person showed stronger relationships with peak loading when daily sewage flow per person was splitted into two ranges. Population density (i.e., drainage population divided by drainage area) and daily sewage flow per person considerably were related with peak loading coefficient when daily sewage flow per person is less than about 400 Lpcd.

Prediction Method of Settlement Based on Field Monitoring Data for Soft Ground Under Preloading Improvement with Ramp Loading (점증 재하를 고려한 선행재하 공법 적용 연약지반의 현장 계측을 통한 침하량 예측 방법의 개발)

  • Woo, Sang-Inn;Yune, Chan-Young;Baek, Seung-Kyung;Chung, Choong-Ki
    • Proceedings of the Korean Geotechical Society Conference
    • /
    • 2008.03a
    • /
    • pp.452-461
    • /
    • 2008
  • Previous settlement prediction method based on settlement monitoring such as hyperbolic, monden method were developed under instantaneous loading condition and have restriction to be applied to soft ground under ramp loading condition. In this study, settlement prediction method under ramp loading was developed. New settlement prediction method under ramp loading considers influence factors of consolidation settlement and increase accuracy of settlement prediction using field monitoring data after ramp loading. Large consolidation tests for ideally controlled one dimensional consolidation under ramp loading condition were performed and the settlement behavior was predicted based on the monitoring data. As a result, new prediction method is expected to have great applicability and practicability for the prediction of settlement behavior.

  • PDF

A Quantitative Evaluation of ${\Delta}K_{eff}$ Estimation Methods Based on Random Loading Crack Growth Data. (랜덤하중하의 피로균열진전 데이터를 이용한 ${\Delta}K_{eff}$ 평가법의 정량적 평가)

  • Koo, Ja-Suk;Song, Ji-Ho;Kang, Jae-Youn
    • Proceedings of the KSME Conference
    • /
    • 2004.11a
    • /
    • pp.208-213
    • /
    • 2004
  • Methods for estimation of the effective stress intensity factor range (${\Delta}K_{eff}$) are evaluated for narrow and wide band random loading crack growth test data of 2024-T351 aluminum alloy. Three methods of determining $K_{op}$, visual measurement, ASTM offset compliance method, and the neural network method proposed by Kang and Song, and three methods of estimating ${\Delta}K_{eff}$, conventional, the 2/PI0 and 2/PI methods proposed by Donald and Paris, are compared in a quantitative manner by using the results of fatigue crack growth life prediction under random loading. For all $K_{op}$ determination methods discussed, the 2/PI0 and 2/PI methods of estimating ${\Delta}K_{eff}$ provide better results than conventional method for narrow and wide band random loading data.

  • PDF

Assessment of whipping and springing on a large container vessel

  • Barhoumi, Mondher;Storhaug, Gaute
    • International Journal of Naval Architecture and Ocean Engineering
    • /
    • v.6 no.2
    • /
    • pp.442-458
    • /
    • 2014
  • Wave induced vibrations increase the fatigue and extreme loading, but this is normally neglected in design. The industry view on this is changing. Wave induced vibrations are often divided into springing and whipping, and their relative contribution to fatigue and extreme loading varies depending on ship design. When it comes to displacement vessels, the contribution from whipping on fatigue and extreme loading is particularly high for certain container vessels. A large modern design container vessel with high bow flare angle and high service speed has been considered. The container vessel was equipped with a hull monitoring system from a recognized supplier of HMON systems. The vessel has been operating between Asia and Europe for a few years and valuable data has been collected. Also model tests have been carried out of this vessel to investigate fatigue and extreme loading, but model tests are often limited to head seas. For the full scale measurements, the correlation between stress data and wind data has been investigated. The wave and vibration damage are shown versus heading and Beaufort strength to indicate general trends. The wind data has also been compared to North Atlantic design environment. Even though it has been shown that the encountered wind data has been much less severe than in North Atlantic, the extreme loading defined by IACS URS11 is significantly exceeded when whipping is included. If whipping may contribute to collapse, then proper seamanship may be useful in order to limit the extreme loading. The vibration damage is also observed to be high from head to beam seas, and even present in stern seas, but fatigue damage in general is low on this East Asia to Europe trade.

A Research on Development of Bills of Material Using Web Grid for Product Lifecycle Management

  • Yoo, Ji-Hyun
    • Journal of the Korea Society of Computer and Information
    • /
    • v.22 no.12
    • /
    • pp.131-136
    • /
    • 2017
  • PLM(Product Lifecycle Management) is an information management system that can integrate data, processes, business systems and human resources throughout the enterprise. BOM(Bills Of Material) is key data for designing, purchasing materials, manufacturing planning and management, which is basic for product development throughout the product life cycle. In this paper, we propose the efficient system to increase the data loading speed and the processing speed when using such large BOM data. We present the performance and usability of IMDG (In Memory Data Grid) for data processing when loading large amounts of data. In the UI, using the pure web grid of JavaScript instead of the existing data loading method can be improve performance of data managing.

MEAN LOAD EFFECT ON FATIGUE OF WELDED JOINTS USING STRUCTURAL STRESS AND FRACTURE MECHANICS APPROACH

  • Kim, Jong-Sung;Kim, Cheol;Jin, Tae-Eun;Dong, P.
    • Nuclear Engineering and Technology
    • /
    • v.38 no.3
    • /
    • pp.277-284
    • /
    • 2006
  • In order to ensure the structural integrity of nuclear welded structures during design life, the fatigue life has to be evaluated by fatigue analysis procedures presented in technical codes such as ASME B&PV Code Section III. However, existing fatigue analysis procedures do not explicitly consider the presence of welded joints. A new fatigue analysis procedure based on a structural stress/fracture mechanics approach has been recently developed in order to reduce conservatism by erasing uncertainty in the analysis procedure. A recent review of fatigue crack growth data under various mean loading conditions using the structural stress/fracture mechanics approach, does not consider the mean loading effect, revealed some significant discrepancies in fatigue crack growth curves according to the mean loading conditions. In this paper, we propose the use of the stress intensity factor range ${\Delta}K$ characterized with loading ratio R effects in terms of the structural stress. We demonstrate the effectiveness in characterizing fatigue crack growth and S-N behavior using the well-known data. It was identified that the S-N data under high mean loading could be consolidated in a master S-N curve for welded joints.

Crack Closure and Growth Behavior of Short Fatigue Cracks under Random Loading (Part II : Growth Behavior and Growth Life Prediction) (짧은 피로균열의 랜덤하중하의 균열닫힘 및 진전거동(Part II : 진전거동 및 진전수명예측))

  • Lee, Shin-Young;Song, Ji-Ho
    • Proceedings of the KSME Conference
    • /
    • 2000.11a
    • /
    • pp.141-146
    • /
    • 2000
  • Crack closure and growth behavior of physically short fatigue cracks under random loading are investigated by performing narrow- and wide-band random loading tests for various stress ratios. The importance of the crack closure phenomenon is examined by predicting the growth lives of short cracks using obtained crack opening behavior. Artificially prepared two-dimensional, short through-thickness cracks are used. The crack opening load of short cracks is much lower under random loading than under constant-amplitude loading corresponding to the largest load cycle in a random load history. This result indicates that the largest load cycle in a random load history has an effect to enhance crack opening of short cracks. Most of the life prediction ratios are within the factor of 2 scatter band except several data at very short crack sizes, indicating that crack growth predictions based on the measured crack opening data are excellent. From the results obtained in this study, it can be concluded that crack closure is the primary factor governing fatigue crack growth of short cracks under random loading as well as under constant-amplitude loading.

  • PDF