Teragen crosscheck of master / HDDS-3053 / HDDS-2717

Test method

• 100G Teragen has been executed on Yarn with using o3fs as input / output dir
• 3 nodes are used for HDFS/Ozone 3 nodes for YARN
• Storage
  • Ratis log is saved to memdisk
  • OM, SCM metadata is saved to disk (emptyDir)
  • Chunks are saved to real disk (hostPath)
• Default settings used except:
  • GC
    • HADOOP_OPTS: -server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly
  • Handlers:
    • OZONE-SITE.XML_ozone.om.handler.count.key: “250”
    • OZONE-SITE.XML_ozone.scm.handler.count.key: “250”

Test script:

ROWS=$(numfmt --from=auto --to-unit=100 100G)
OUTPUT_DIR=teragen-$(shuf -i 1000-2000 -n 1)

#for HDFS test it was commented out
OUTPUT_DIR=o3fs://bucket1.vol1.ozone-om-0.ozone-om/$OUTPUT_DIR

MR_EXAMPLES_JAR=/opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar

time yarn jar $MR_EXAMPLES_JAR teragen \
-Dmapreduce.map.log.level=INFO \
-Dmapreduce.reduce.log.level=INFO \
-Dyarn.app.mapreduce.am.log.level=INFO \
-Dmapreduce.map.cpu.vcores=1 \
-Dmapreduce.map.java.opts=-Xmx1536m \
-Dmapreduce.map.maxattempts=1 \
-Dmapreduce.map.memory.mb=2048 \
-Dmapreduce.map.output.compress=true \
-Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.Lz4Codec \
-Dmapreduce.reduce.cpu.vcores=1 \
-Dmapreduce.reduce.java.opts=-Xmx1536m \
-Dmapreduce.reduce.maxattempts=1 \
-Dmapreduce.reduce.memory.mb=2048 \
-Dyarn.app.mapreduce.am.command.opts=-Xmx768m \
-Dyarn.app.mapreduce.am.resource.mb=1024 \
-Dmapreduce.task.io.sort.factor=100 \
-Dmapreduce.task.io.sort.mb=384 \
-Dmapred.map.tasks=92 \
-Dio.file.buffer.size=131072 \
$ROWS $OUTPUT_DIR

sleep 10000000

Environment (margo-01)

• 6 Dell PowerEdge R430
• e5-2630 v4 @ 2.2Ghz
• 128GB Ram
• 4-2TB disks (xfs)

Provisioned with kubernetes. 3 nodes are assigned to HDFS and Ozone, 3 nodes are assigned to Yarn

Results

Running teragen on master

• docker image: elek/ozone-dev:a68ed78d9
• branch: master
• date: 2020-03-04

Running teragen with master + using 10 chunk writer thread (=HDDS-3053)

Same as before but dfs.container.ratis.num.write.chunk.threads set to 10

Running teragen with HDDS-2717

• docker image: elek/ozone-dev:980b7cfb7
• branch: elek/HDDS-2717
• date: 2020-03-04

Stats

Statistics after one (Ozone master) teragen with 100G write